PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges

Christian Arzate Cruz, Takeo Igarashi

arXiv:2105.12949v1cs.HC

TL;DR

Standard ML methods can be insufficient for complex, impactful applications, and interactive RL has not been widely adopted in HCI. This survey equips HCI researchers with technical background, feedback approaches, and design principles; it reports benefits of human guidance while noting constraints in feedback and reward design.

  • Problem

    Standard ML methods can be insufficient for complex, impactful applications, and interactive RL remains underused in HCI.

    Source: INTRODUCTION · p. 1

  • Method

    The paper surveys interactive RL, describing feedback approaches and proposing generic design principles for applications.

    Source: INTRODUCTION · p. 1

  • Results

    The survey reports that human guidance can make exploration natural and sample-efficient, while too few studies support conclusive comparison of value-function augmentation.

    Source: Exploration Process · p. 4, A reward function · p. 5

  • Takeaways & Limitations

    The proposed design principles guide implementation, and the survey highlights feedback approaches for personalization, complex-task learning, and exploration guidance.

    Source: INTRODUCTION · p. 1

    Human feedback may arrive after the relevant actions and refer to a single state or a sequence; deficient reward design can produce nonoptimal reward-hacking behavior.

    Source: Design Dimensions · p. 3

Abstract

from arXiv · show

Interactive reinforcement learning (RL) has been successfully used in various applications in different fields, which has also motivated HCI researchers to contribute in this area. In this paper, we survey interactive RL to empower human-computer interaction (HCI) researchers with the technical background in RL needed to design new interaction techniques and propose new applications. We elucidate the roles played by HCI researchers in interactive RL, identifying ideas and promising research directions. Furthermore, we propose generic design principles that will provide researchers with a guide to effectively implement interactive RL applications.

INTRODUCTION

Interactive RL may help address challenges that standard ML methods face in complex applications, yet it remains underused in HCI. This survey gives HCI researchers technical background, reviews feedback approaches, and proposes design principles for interactive RL.

INTRODUCTION

  • HCI researchers can contribute by designing interaction techniques and applications, supported here by a survey of feedback approaches and generic design principles.The principles are intended to guide implementation across physical and simulated environments.Source: INTRODUCTION · p. 1, Scope of this Paper · p. 2

Scope of this Paper

  • The survey focuses on interactive RL work in robotics, HCI, and Game AI published from 2010 to 2019, selected to support generic design rules.It narrows a broader literature to approaches applied in human-in-the-loop RL.Source: Scope of this Paper · pp. 1–2

REINFORCEMENT LEARNING (RL)

  • An MDP models an agent acting in a stochastic environment through states, actions, transition probabilities, rewards, and a discount factor.The reward function specifies goals, while the transition function captures the effects of actions in particular states.Source: REINFORCEMENT LEARNING (RL) · p. 2

INTERACTIVE REINFORCEMENT LEARNING

Interactive RL places a human in the loop to tailor parts of an RL algorithm, with feedback channels that guide its components. The survey illustrates this approach through varied testbeds spanning HCI, Game AI, and robotics.

INTERACTIVE REINFORCEMENT LEARNING

INTERACTIVE RL TESTBEDS

  • The survey’s testbeds range from GridWorld and Pac-Man to Mario AI, robots, and Sophie’s kitchen, spanning HCI, Game AI, and robotics settings.GridWorld supports fast, simple experiments; the robot platforms support natural interaction, while Sophie’s kitchen explores showing uncertainty to users.Source: INTERACTIVE RL TESTBEDS · pp. 2–3, Figure 2 · p. 3

Applications

Interactive RL has applications including robot dancing and adaptable dialogue systems, and the paper analyzes how human feedback tailors RL algorithms’ low-level modules. It uses that analysis to develop design guidance for interactive RL.

Applications

  • Interactive RL applications include teaching robots to dance, creating adaptable task-oriented dialogue systems, and generating dialogue.The survey also identifies possible adaptations of automatic RL for procedural game content, music generation, and modeling social influences in bots.Source: Applications · p. 3

DESIGN GUIDES FOR INTERACTIVE RL

  • The paper analyzes how human feedback tailors RL algorithms’ low-level modules, calling these interaction forms design dimensions and using a classification to compare them.The section aims to provide HCI researchers with foundations for designing novel interaction techniques.Source: Design Dimensions · p. 3, DESIGN GUIDES FOR INTERACTIVE RL · p. 3

Design Dimensions

  • Reward shaping iteratively adjusts rewards to encode desired behaviors and balance goals; a common form adds a human-designed shaping reward F to environmental reward R.Hand-engineered reward functions have been demonstrated to increase learning rates, while other approaches use human feedback as the agent’s sole reward source.Source: Design Dimensions · p. 3
  • Reward shaping can supply information when environmental rewards are absent or sparse, including in highly stochastic environments.Human feedback may apply to past actions after a delay, creating a credit-assignment problem involving one state or a sequence of states.Source: Design Dimensions · p. 3
  • Deficient reward designs can produce reward hacking: agents may pursue high-reward behaviors that fail to achieve designers’ goals when goal trade-offs were not anticipated.In one cited boat-racing example, the agent pursued special targets rather than moving toward the goal.Source: Design Dimensions · p. 3

Policy

The section contrasts policy shaping, which uses action advice to update behavior, with exploration guidance that steers agents toward useful actions. These approaches can depend on expert knowledge, environment access, or short-horizon advice.

Policy

  • Binary critique approaches that prune a bad action from the action set bias action selection rather than the policy, so the paper categorizes them as guided exploration.Source: Policy · p. 4
  • Policy shaping uses action advice to update an agent’s behavior and requires access to an expert user’s partial policy for the task.Unlike reward-based representation, it does not depend on expressing the problem through a reward function, but improving learning assumes the user knows a near-optimal policy.Source: Policy · p. 4

Exploration Process

  • Effective RL policies balance exploitation of actions that worked before with exploration of untried actions.Source: Exploration Process · p. 4
  • Guided exploration injects human knowledge to steer agents toward high-reward states, avoiding actions that do not correspond to an optimal policy.The approach assumes users understand the task well enough to identify at least one near-optimal policy; the survey describes evidence for it as a natural and sample-efficient interactive RL approach.Source: Exploration Process · p. 4
  • When users point to an object, exploration can direct actions toward it, requiring an environment model to determine how to reach the suggested object.Source: Exploration Process · p. 4
  • Myopic agents choose each action from the user’s immediate advice without considering long-term effects, and this guidance approach tends to produce overfitting.It has nevertheless been used successfully in different applications.Source: Exploration Process · pp. 4–5

Value Function

The section discusses value-function augmentation and low-level design alternatives, including function approximation and hierarchical decomposition. It also notes differences in reported user-experience evidence across these approaches.

A reward function

  • A value function estimates expected future reward from each state under the current policy, and computing one is a necessary intermediate step toward finding a policy.Source: A reward function · p. 5
  • Combining a human-feedback value function with the agent’s value function has been shown to bias agent behavior and accelerate learning.There are too few studies to compare this design dimension conclusively with others, though reusing an expert value function across similar state-action spaces might reduce required human feedback.Source: A reward function · p. 5

Design Dimension Alternatives

  • Function approximation supports continuous state spaces and can complement other interaction channels; participant-designed approximators accelerated learning in soccer but worsened performance in Super Mario Bros.The survey suggests effective function approximation may be harder to design than reward shaping in complex environments.Source: Design Dimension Alternatives · p. 5
  • Hierarchical reinforcement learning decomposes high-dimensional state-action spaces into smaller subproblems or temporally extended actions, but the survey reports no user-experience studies in interactive RL.An expert-defined hierarchy can provide skills reusable across applications.Source: Design Dimension Alternatives · p. 5

Types of Feedback

  • The next section describes common human-feedback types and how each delivers user expertise to human-knowledge integration methods.Source: Types of Feedback · p. 5

Binary critique

This section surveys feedback methods that map human knowledge onto interactive RL, from critique and advice to goal guidance. It describes their mechanisms and reports that guidance and action advice can improve learning or user experience in particular settings.

Binary critique

  • Binary critique asks whether the agent’s last action was satisfactory, but unpredictable signals and users stopping after learning can make it suboptimal.Source: Binary critique · p. 5
  • Hand-engineered heuristic functions map state-action pairs to positive or negative critiques to reduce human feedback, though their viability in complex environments remains to be tested [85].Source: Binary critique · p. 5
  • Query-based inverse reward design presents sub-rewards designed to reveal unknown environmental rewards and distinguish among suboptimal alternatives [38] [76].Source: Binary critique · pp. 5–6

Scalar-valued critique

  • Scalar-valued critique uses the magnitude of users’ evaluations to learn a reward function, but people have been found to provide nonoptimal numerical signals [60].Source: Scalar-valued critique · p. 6

Action Advice

  • Action advice lets users supply the action they consider optimal at a state; its immediate effect on policy was reported as less frustrating.Source: Action Advice · p. 6

Guidance

  • User guidance directs exploration toward user-specified goals, and experiments found that exploration guides alone produced the best results while reducing required feedback.Source: Guidance · p. 6

RECENT RESEARCH RESULTS

Reward Shaping

This section reviews reward- and policy-shaping approaches that incorporate human feedback into RL. Reported findings include feedback-dependent learning benefits, practical constraints on heuristic functions, and a user-experiment comparison favoring action advice over critique.

Reward Shaping

  • Negative numerical rewards can prompt an agent to undo its last action, revealing learning progress and motivating more negative feedback; this achieved good performance with less feedback.Source: Reward Shaping · p. 6
  • Heuristic functions can inject feedback automatically, but they are difficult to build, require programming skills, and lack well-studied natural tools for non-experts to create ones that generalize [85].Source: Reward Shaping · p. 6
  • Deep TAMER weights user rewards for individual state-action pairs, while DQN-TAMER combines TAMER variants with environmental and binary human feedback to shape reward [108] [10].Source: Reward Shaping · p. 6

Policy Shaping

  • Bayesian policy shaping infers a user policy from state-action critiques and reportedly outperformed reward shaping, but experiments used a simulated oracle rather than human users.Source: Policy Shaping · p. 7
  • Action advice required less training time, performed objectively better, and yielded a better user experience than critique in the reported comparison [58].Source: Policy Shaping · p. 7
  • Deep COACH uses raw pixels, but its evaluation used simplistic toy problems, leaving its suitability for real scenarios uncertain [12] [71].Source: Policy Shaping · p. 7

Guided Exploration Process

This section reviews exploration guidance and augmented value functions as ways to use human knowledge in interactive RL. Experiments report that user-only exploration guides performed best in one setting, while value-function reuse offers transfer without requiring an environment model.

Guided Exploration Process

  • In experiments, using only user exploration guides produced the best results and reduced the amount of feedback needed.Source: Guided Exploration Process · p. 7
  • Guidance appears effective for performance and user experience, but requires an environment model that may be difficult to create in complex environments.Source: Guided Exploration Process · p. 7

Augmented Value Function

  • Human and agent value functions can be combined; HAT uses recorded human traces to shape the agent’s q-values with a bonus for aligned state-action pairs.Source: Augmented Value Function · p. 7
  • Augmented value functions support transfer between users and combination of feedback sources across states, without requiring an environment model.Source: Augmented Value Function · p. 8

Inverse Reward Design

This section explains inverse reward design and surveys design considerations for interactive RL feedback. It reports reduced reward hacking with inverse reward design and describes timing, fatigue, active queries, and feedback propagation as relevant to performance and user effort.

Inverse Reward Design

  • Inverse reward design infers a distribution over true rewards from a proxy reward, environment model, and designer behavior; experiments found that it lessened reward-hacking problems [38].Source: Inverse Reward Design · p. 8
  • Inverse reward design begins with reward shaping and then queries users about preferences in high-uncertainty states, motivating interaction techniques for explaining uncertainty and debugging models.Source: Inverse Reward Design · p. 8

Feedback

  • Small changes in the expected timing distribution of user feedback can greatly affect Deep TAMER performance, making expert timing knowledge important [108].Source: Feedback · p. 8
  • Users often reduce feedback quantity and positivity over time, and the amount and valence of feedback can depend on the agent’s behavior [71].Source: Feedback · p. 8
  • Gamification has been shown to improve feedback quantity and quality, while active questioning can achieve good performance with less demonstrator attention [32].Source: Feedback · p. 8
  • Propagating feedback to similar states with an approximation function has been effective against sample inefficiency in complex environments such as videogames [108].Source: Feedback · pp. 8–9

Typification of the End-user

The section characterizes end-users by their task knowledge, teaching preferences, and safety concerns, then connects those traits to feedback and design choices. It also describes evaluation and explanation techniques intended to make interactive teaching cycles more efficient and legible.

Typification of the End-user

  • End-users differ in task knowledge, preferred teaching strategy, and safety concerns; high-severity errors can reduce willingness to interact and affect feedback quality and quantity.Task knowledge also affects feedback quality, quantity, and delay, and may limit demonstrations or action advice.Source: Typification of the End-user · p. 9
  • End-user typification can guide feedback and design-dimension choices; exploration is reported as the most economical dimension, while myopic policy-design agents succeed in some tasks.Some dimensions can better handle non-optimal human policies, although the dimensions assume users know a policy good enough to solve the task.Source: Typification of the End-user · p. 9
  • Hierarchical decomposition and function approximation need users with deep problem understanding; combining dimensions or actively selecting one could reduce required feedback.A function-approximation and policy-design combination is reported to achieve positive videogame results with 15 minutes of human feedback.Source: Typification of the End-user · p. 9

Fast Interaction Cycles

  • Fast behavior evaluation can shorten iterative teaching loops; proposed approaches include reward-function queries, trajectory summaries, and crowd evaluation.Users select among reward functions, agents can present multiple trajectories for summarization, and evaluation can be distributed among users.Source: Fast Interaction Cycles · p. 9
  • HIGHLIGHTS summarizes behavior through key-state trajectories, ranking a state by the gap between its highest and lowest expected returns.A state is important when an action can substantially decrease expected reward.Source: Fast Interaction Cycles · p. 9
  • Explanations can clarify an agent’s decision process; gaze can communicate uncertainty, and users gave more feedback when the agent displayed it.Gaze indicates promising options, with shifts among objects signaling indecision and allowing time for user feedback.Source: Fast Interaction Cycles · p. 9

Design Implications

The section identifies application-design challenges spanning user accessibility, high-dimensional environments, evaluation, generalization, and simulated feedback. It argues that addressing these areas is important for practical interactive RL across users and settings.

Design Implications

  • Adaptive systems and demonstrations for complex robot tasks are promising directions for making interactive RL accessible to non-experts and users with physical limitations.Demonstrations can be difficult for users to provide when they cannot show an agent how to perform the task.Source: Design Implications · p. 9

High-dimensional Environments

  • Interactive RL in high-dimensional environments remains challenging, and broadening useful domains is presented as important for extending machine learning’s impact in people’s lives.Autoencoders that spread feedback to similar states perform well in high-dimensional settings; crowd-sourced feedback or reward construction is underexploited.Source: High-dimensional Environments · pp. 9–10

Lack of Evaluation Techniques

  • HCI evaluations are limited and often prioritize algorithm performance over interaction experience; results from simple testbeds may not generalize consistently to complex ones.Reward shaping was effective and easy to use in GridWorld but challenging and ineffective for Infinite Mario, even for experts.Source: Lack of Evaluation Techniques · p. 10

Lack of Human-like Oracles

  • Simulated oracles can save time early in development, but their results can differ from human-user results because they do not behave like humans.Further studies are needed to identify what distinguishes human feedback from simulated-oracle feedback.Source: Lack of Human-like Oracles · p. 10

Modeling Users

The section calls for user models and stronger evidence about how design dimensions and feedback types work for different end-users. It also identifies safety, efficient evaluation, and explainability as open directions, concluding that the survey aims to support HCI researchers designing interactive RL techniques and applications.

Modeling Users

  • A formal user model for interactive RL or IML has not been proposed; one could support help requests timed to avoid fatigue and interaction adapted to user preferences.Mapping user types to preferred interaction channels would require empirical studies.Source: Modeling Users · p. 10

Combining Different Design Dimensions

  • Understanding strengths and weaknesses across design dimensions and feedback types requires user-experience studies across their combinations and testbeds with small and large state spaces.Mapping those choices to end-user type, together with an end-user model, could support applications that adapt to the current user.Source: Combining Different Design Dimensions · p. 10

Safe interactive RL

  • Safe interactive RL needs ways to restrict dangerous, unwanted, or biased behavior; Safe RL techniques and interaction methods for imposing limits are proposed starting points.The cited Safe RL techniques aim to maintain performance while respecting safety constraints during learning and deployment.Source: Safe interactive RL · p. 10

Fast Evaluation of Behaviors

  • Visualization-based behavior summaries and uncertainty explanations remain understudied approaches intended to reduce evaluation time and help users give higher-quality feedback.The approaches include summarizing behavior and explaining uncertainty.Source: Fast Evaluation of Behaviors · p. 10

Explainable interactive RL

  • Explainable interactive RL aims to help people understand an agent’s state representation and policy reasoning, while transparent models could help users identify errors or unwanted bias.Debugging decisions in particular states is described as important for applications with substantial effects on people’s lives.Source: Explainable interactive RL · p. 10

CONCLUSIONS

  • The paper presents its survey as technical background for HCI researchers designing new interaction techniques and applications, and as a step toward wider HCI use of interactive RL.Source: CONCLUSIONS · p. 10
Loading 2105.12949v1…