PaperlayerView exact revision on arXiv

Source-linked AI summary

Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

William Saunders, Girish Sastry, Andreas Stuhlmueller, Owain Evans

arXiv:1707.05173v1cs.AIcs.LGcs.NE

TL;DR

The paper asks how model-free reinforcement-learning agents can explore without catastrophic actions, given that trial-and-error learning may require trying harmful actions. It formalizes human intervention through HIRL and trains a supervised learner to imitate the human’s blocking decisions. HIRL prevents simple catastrophes without impairing learning, but complex catastrophes and human labor costs limit the approach’s scalability.

  • Problem

    Model-free reinforcement learning can require trialing a catastrophic action before learning to avoid it, making human oversight necessary for zero-catastrophe training.

  • Method

    HIRL places a human between the RL agent and environment, then trains a supervised Blocker to imitate the human’s intervention policy.

  • Results

    HIRL prevented all catastrophes for simple classes in Pong and Space Invaders, reduced but did not eliminate them in Road Runner, and did not impair learning relative to Reward Shaping.

  • Takeaways & Limitations

    Human intervention can enable zero-catastrophe learning for simple catastrophe classes, whereas reward-based RL alone failed because of catastrophic forgetting.

  • Takeaways & Limitations

    HIRL’s implementation does not scale to complex environments because near-perfect Blocker reliability, human labeling effort, and sufficiently slow interaction can be infeasible.

Abstract

from arXiv · show

AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a single catastrophe.

1 Introduction

The paper proposes HIRL, a human-intervention scheme for reinforcement learning, and evaluates whether supervised imitation can reduce oversight labor while preventing catastrophes. Results show success on simple Atari catastrophes, partial success on complex ones, and serious scalability challenges.

  • Approach: HIRL lets a human block catastrophic actions while training a supervised learner to imitate those intervention decisions and take over oversight.The scheme is designed to reduce the human labor required to supervise reinforcement-learning agents.
  • Evaluation: For the first 4.5 hours, a human watched every frame and intervened to block catastrophic actions in Pong, Space Invaders, and Road Runner.The Atari evaluation used three games to investigate HIRL’s scalability.
  • Comparison: An RL-only baseline using large negative catastrophe rewards never stopped causing catastrophes, with Pong failures attributed to catastrophic forgetting.HIRL therefore succeeded where the reward-based safety approach failed in this evaluation.
  • Challenges: HIRL faces adversarial distribution shift and infeasible human time-costs as environments become more complex.The Road Runner CNN was not robust to an adversarial agent, while other Atari games could require years of human oversight.

2 HIRL: A Scheme for Safe RL via Human Intervention

HIRL defines safety through human-specified catastrophic actions and keeps a human or learned Blocker between the RL agent and environment. The scheme replaces unsafe actions, penalizes them, and trains a supervised Blocker to continue oversight, while recognizing distribution shift and scalability constraints.

  • 2.1 Motivation for HIRL: HIRL defines an RL agent as safe when it never takes catastrophic actions during training, with the human overseer specifying which actions are unacceptable.The paper distinguishes catastrophic actions from merely sub-optimal actions.
  • 2.1 Motivation for HIRL: Model-free RL alone cannot guarantee zero catastrophes because trial-and-error learning generally requires trying an unknown catastrophic action before avoiding it.The paper focuses on model-free RL; model-based algorithms may have advantages for avoiding catastrophes.
  • 2.2 Formal Specification of HIRL: During HIRL, a human observes each state-action pair, replaces catastrophic actions with safe actions, and substitutes a penalty for the resulting reward.This intervention occurs before the catastrophic action reaches the environment.
  • 2.2 Formal Specification of HIRL: The Blocker learns from human state-action labels, then takes over continuous oversight after achieving good held-out performance.The Blocker is trained as a supervised classifier and can be reused with different RL agents.
  • 2.3 When is HIRL feasible?: Blocker reliability is challenged by shifting state-action distributions, adversarial examples, and the potentially huge human labor required for near-perfect recognition.Multiple Human Oversight phases are one proposed response to distribution shift.

3.1 Design of Experiments and Implementation of HIRL

The experiments define locally avoidable Atari catastrophes, collect human interventions, train a CNN Blocker, and compare HIRL with reward shaping and no oversight. The implementation uses staged human and Blocker oversight, with reward-penalty details differing for clipped-reward Space Invaders.

  • 3.1 Design of Experiments and Implementation of HIRL: The experiments use Atari environments with A3C and Double DQN agents, interactive action blocking, and reward clipping only for Double DQN.The Blocker is a CNN trained on full Atari images without pooling layers.
  • 3.1 Design of Experiments and Implementation of HIRL: The HIRL procedure comprises 4.5 hours of slowed human oversight, CNN Blocker training with a low-false-negative threshold, and 12-24 hours of full-speed Blocker oversight.The human records blocking decisions during the first phase, and the Blocker then replaces the human.
  • 3.1.1 What are Catastrophes in Atari?: The study treats Pong paddle descent, Space Invaders barrier shooting, and Road Runner Level 1 death as catastrophic outcomes.These outcomes are selected as locally avoidable, so an overseer can intervene shortly before catastrophe without game-playing skill.
  • 3.1.2 Baseline: Human-trained Reward Shaping: The Blocker blocks catastrophic actions and replaces them with safe actions, while the Reward Shaping baseline applies negative rewards without blocking.The baseline cannot achieve zero catastrophes because it must try catastrophic actions to learn their negative reward.
  • 3.1.2 Baseline: Human-trained Reward Shaping: No Oversight agents experience more than ten thousand catastrophes in each game, showing that preventing catastrophes is not automatic.Figure 3 reports cumulative catastrophes over time with mean and standard error.
  • 3.1.2 Baseline: Human-trained Reward Shaping: The Reward Shaping baseline has fewer catastrophes than No Oversight, but its catastrophe rate does not converge to zero and is linked to catastrophic forgetting in Pong.For Space Invaders, reward clipping prevents setting a penalty larger than the total discounted return.

3.2 Summary of Results

HIRL prevents catastrophes more reliably than reward shaping in some Atari games without harming learning, but performance depends on catastrophe complexity and can still require substantial human oversight.

  • More than 10,000 catastrophes occur in each game without human oversight.
  • Extreme reward outliers remain difficult for Deep RL algorithms to handle correctly, motivating possible DQN modifications.
  • HIRL achieves zero catastrophes with impressive performance in Pong and Space Invaders, but not in Road Runner.
  • Reward shaping lowers total catastrophes relative to no oversight, but catastrophe rates do not converge to zero.
  • Catastrophic forgetting causes persistent catastrophes in Pong under reward shaping.
  • HIRL does not show negative learning effects relative to reward shaping and appears to improve faster with better overall reward.

3.3 Pong: Detailed Analysis of the Blocker and of Human Time Cost

Pong experiments show that HIRL can generalize a learned Blocker across agents and prevent catastrophes, while detailed analysis links residual attempted catastrophes to catastrophic forgetting.

  • HIRL enables an A3C agent to master Pong without incurring catastrophes.
  • The Blocker generalizes across A3C variants, Double DQN, and a catastrophe-loving A3C agent while blocking all catastrophes without preventing mastery.
  • Reward shaping initially uses human-blocked examples, then applies negative rewards without blocking catastrophic actions.
  • Measuring blocked catastrophic attempts estimates how many catastrophes would occur if blocking were removed.
  • HIRL attempts catastrophes at about 10^-5 per frame or 10^-2 per episode in Pong.
  • The catastrophe rate decreases only when the learning rate is zero, strongly suggesting catastrophic forgetting rather than policy stochasticity.

3.4 Road Runner: Blocking an Adversarial Agent

In Road Runner, the Blocker initially prevented the usual Score Exploit but later failed when the agent discovered a visually distinct way to die. Correcting training labels reduced deaths substantially, but did not eliminate them.

  • Catastrophe definition: Road Runner treats losing a life on Level 1 as catastrophic, although touching the Coyote is easily avoidable by running away.The game rewards eating birdseed and knocking out the Coyote.
  • Adversarial failure: The agent’s random exploration can find catastrophic actions that differ from the Blocker’s training examples, creating adversarial examples for its CNN.The usual Score Exploit was blocked, but a new exploit emerged.
  • Adversarial failure: The agent discovered an alternative Score Exploit after 16 million frames by waiting at the screen’s top-right corner for the Coyote to kill it.This position was visually distinct and presumably fooled the Blocker CNN.
  • Early Blocker behavior: Before the Score Exploit appeared, several catastrophe-rate spikes indicated additional Blocker failures and were anti-correlated with reward.These spikes did not indicate a Score Exploit.
  • Label correction: After correcting labeling mistakes in 20,000 training frames, the average death rate fell 20-fold from 0.002 to 0.0001 deaths per frame.The No Oversight baseline had a death rate of 0.005.

4 Challenges in Scaling Up HIRL

Scaling HIRL depends on the human time needed to label enough state-action observations, which can become infeasible when catastrophes are rare or difficult to encounter. The paper estimates this cost analytically and illustrates the problem in Pong and Montezuma’s Revenge.

  • Human time-cost: Human time-cost is C = thuman × Nall, where thuman is time per human label and Nall is the number of labeled observations.The training set records the agent’s observations and whether each state-action pair is catastrophic.
  • Human time-cost: Reducing Nall is difficult because the Blocker needs substantial positive and negative examples to achieve an extremely low false-negative rate.Increasing attempted catastrophes can reduce the proportion of safe observations without harming Blocker performance.
  • Human time-cost: The cost can also be written C = thuman × ρ × Ncat, where ρ is the ratio of all observations to catastrophe observations and Ncat is the number of labeled catastrophes.Rare catastrophes increase ρ and therefore increase human labor.
  • Pong: In Pong, the oversight phase lasted 4 hours, with thuman = 0.8s, ρ = 166, and Ncat = 120.These values instantiate the human time-cost formula.
  • Pong: If a pre-trained Pong agent began with ρ = 10^5, labeling would take 110 days despite the catastrophe being simple.The estimate uses 0.8 × 10^5 × 120.
  • Montezuma’s Revenge: Reaching distinct ledges in Montezuma’s Revenge could require 100 million frames, corresponding to at least a year of human oversight.This extrapolation indicates that the implementation would not scale to such environments.

5 Discussion

HIRL prevented simple catastrophes in Pong and Space Invaders but only partially blocked complex catastrophes, while extrapolations showed infeasible human time-costs. The discussion identifies robustness and efficiency techniques needed for broader scaling.

  • Discussion: HIRL blocked all catastrophes in Pong and Space Invaders but only partially blocked more complex catastrophes, and its human time-cost became infeasible in harder environments.The implementation would not scale to other Atari games requiring years of human time.
  • Reducing human labor: More data-efficient Blockers could reduce human cost by reducing Ncat, the number of catastrophe examples needed for training.This directly targets the human-labeling term in the time-cost formula.
  • Reducing human labor: More data-efficient RL could reduce the time needed to encounter the full range of catastrophes, although it can also increase cost by making catastrophes rare.The Pong example illustrates this countervailing effect.
  • Reducing human labor: Seeking out catastrophes lowers ρ by increasing the proportion of catastrophe observations in the Blocker’s training data.Lower ρ reduces the human time-cost C.
  • Reducing human labor: Active learning could let the human pause when the agent is far from dangerous regions or when the Blocker is reliable in a visited region.The human would be queried near dangerous or novel state-space regions.
  • Safety requirement: An oversight-query algorithm must have no false negatives: novel catastrophes must be blocked directly or exposed to human oversight.This requirement preserves the safety guarantee when selectively querying the human.

6 Appendix

The appendix specifies the game agents, reward handling, shared training settings, and action-replacement mechanisms used in the experiments. Road Runner used action pruning, while Pong and Space Invaders used fixed safe replacements.

  • Agent architectures: The Pong and Road Runner agents used an A3C architecture with four convolutional layers and a 256-unit LSTM.The input was 42x42x1 grayscale, with Adam optimization and discount factor 0.99.
  • Agent architectures: Space Invaders used Double DQN with convolutional layers, a 512-unit hidden layer, replay buffer size 1,000,000, and discount factor 0.99.Its exploration rate decreased from 1.0 to 0.01 over training.
  • Reward scaling: Reward handling was game-dependent: Pong divided reward by 1.0, Road Runner by 100.0, and Space Invaders clipped reward to ±1.These were the specified reward transformations for the three games.
  • Blocker training: Shared settings included dropout probability 0.5, balanced positive and negative example weights, and manual label cleaning.Labels were reviewed across episodes and individual frames where the Blocker disagreed with the assigned label.
  • Action replacement: The Blocker was intended to classify catastrophic actions and select safe replacements, but the experiments simplified replacement learning.The implementation used fixed action replacement or action pruning instead of learning the full replacement policy.
  • Action replacement: Fixed Action Replacement substitutes a human-specified safe action, whereas Action Pruning blocks the action and makes the agent choose again.For pruning, the lowest-logit action is never blocked so at least one action remains available.
  • Game-specific techniques: Pong replaced blocked actions with Up, Space Invaders removed Fire, and Road Runner used Action Pruning.These were the game-specific replacement techniques.
  • Space Invaders: Space Invaders required extended labeling because the agent stayed left of barriers after blocked actions; the authors estimated 70 hours to obtain broader coverage.The training set was augmented to include actions under the middle and right barriers.
Loading 1707.05173v1…