PaperlayerView exact revision on arXiv

Source-linked AI summary

A Conceptual Framework for Externally-influenced Agents: An Assisted Reinforcement Learning Review

Adam Bignold, Francisco Cruz, Matthew E. Taylor, Tim Brys, Richard Dazeley, Peter Vamplew, Cameron Foale

arXiv:2007.01544v2cs.AIcs.LGcs.MA

TL;DR

Reinforcement learning faces scaling challenges in larger, more complex environments, while methods using external information lack a shared basis for comparison. This review proposes a conceptual framework and taxonomy for assisted reinforcement learning, identifying seven features across four processing components and three communication links. The taxonomy is intended to support collaboration, comparability, and further research.

  • Problem

    RL can become impractical to scale as environments grow, and approaches using external information lack a shared understanding of their relationships and characteristics.

    Source: §1 · pp. 2–3

  • Method

    The paper reviews externally influenced RL and proposes an assisted reinforcement learning framework and taxonomy describing the functions and links through which external information supports learning and decision-making.

    Source: §2.2 · p. 4, §6 · p. 22

  • Results

    The proposed taxonomy identifies seven key features of assisted RL, organized into four processing components and three communication links.

    Source: §6 · p. 22

  • Takeaways & Limitations

    The authors state that a standardized taxonomy can foster collaboration, comparability, precise descriptions of new approaches, and identification of research questions.

    Source: §1 · p. 3

    For real-time deployment, assistance may become unnecessary or incorrect if the external source cannot observe and respond to the agent quickly enough.

    Source: §5.5 · p. 21

Abstract

from arXiv · show

A long-term goal of reinforcement learning agents is to be able to perform tasks in complex real-world scenarios. The use of external information is one way of scaling agents to more complex problems. However, there is a general lack of collaboration or interoperability between different approaches using external information. In this work, while reviewing externally-influenced methods, we propose a conceptual framework and taxonomy for assisted reinforcement learning, aimed at fostering collaboration by classifying and comparing various methods that use external information in the learning process. The proposed taxonomy details the relationship between the external information source and the learner agent, highlighting the process of information decomposition, structure, retention, and how it can be used to influence agent learning. As well as reviewing state-of-the-art methods, we identify current streams of reinforcement learning that use external information in order to improve the agent's performance and its decision-making process. These include heuristic reinforcement learning, interactive reinforcement learning, learning from demonstration, transfer learning, and learning from multiple sources, among others. These streams of reinforcement learning operate with the shared objective of scaffolding the learner agent. Lastly, we discuss further possibilities for future work in the field of assisted reinforcement learning systems.

1 Introduction

External information is presented as a way to help reinforcement learning scale to larger, more complex scenarios, but approaches using it lack a shared framework for comparison. The introduction defines assisted reinforcement learning and proposes a framework and taxonomy for describing how external information influences learning.

1 Introduction

  • As environments grow, learning time increases and finding an optimal solution can become impractical, motivating external information to help RL scale.External information can support environment representation, decision-making, or more focused exploration.Source: §1 · p. 2
  • The paper defines external information as information originating outside the agent’s representation of the environment, including demonstrations, advice, prior data, and shaping functions.It defines assisted reinforcement learning as techniques using such information before, during, or after training to improve performance and scale to more complex scenarios.Source: §1 · p. 2
  • Assistance may alter agent weights, state or action spaces, rewards, policies, exploration, or information repositories, and may advise the learner or modify the environment directly.The framework also allows the agent to request advice from the external source.Source: §1 · p. 2, Figure 1 · p. 2
  • Because methods have often been grouped by technique despite limited understanding of their shared characteristics, the paper proposes a framework and taxonomy to describe and compare them.The authors state that a standardized taxonomy can foster collaboration, comparability, precise descriptions, and identification of research questions.Source: §1 · pp. 2–3

2 A Conceptual Framework for Assisted Reinforcement Learning

The section defines assisted reinforcement learning (ARL) and presents a framework for describing how external information reaches and influences an RL agent. The framework underpins a taxonomy intended to improve classification and comparison across methods.

2.1 Assisted Reinforcement Learning

  • ARL supplements environment-derived learning with external information to address the difficulty of scaling RL to large, unbounded environments.The section includes interactive reinforcement learning, learning from demonstration, and transfer learning as examples.Source: §2.1 · pp. 3–4

2.2 Conceptual Framework

  • The framework is designed to improve classification and comparison of externally influenced RL methods, while remaining flexible enough to describe existing and future approaches.Not every ARL approach uses all proposed components and links.Source: §2.2 · p. 4
  • The framework has four processing components: an information source, advice interpretation, an external model, and the assisted agent.The components provide, transform, and store information; sources may include humans, repositories, or other agents.Source: §2.2 · pp. 4–5
  • Three communication links—temporality, advice structure, and agent modification—convey information or constrain where or when it is provided.The framework describes the transmission, modification, and modality of sourced information.Source: §2.2 · pp. 4–5

• Agent modification:

  • Agent modification can alter the environmental reward signal or directly change the agent’s behaviour or decision-making process.The assisted agent must determine how to incorporate the provided information and may choose whether to follow suggested actions.Source: • Agent modification · p. 6

3 Assisted Reinforcement Learning Taxonomy

The ARL taxonomy organizes assistance into four processing components and three communication links, from information sources through interpretation and retention to agent modification. Its subsections describe how advice is timed, represented, stored, and applied while the agent remains autonomous without assistance.

3 Assisted Reinforcement Learning Taxonomy

  • The taxonomy classifies ARL through four processing components and three communication links, treating them as properties and relations in an ontology of ARL problems.The authors use “taxonomy” to classify framework elements, not to classify individual ARL methods.Source: §3 · p. 6

3.1 Information Source

  • Sources may be humans, agents, or digital sources; combining them widens available information but may require managing conflicting advice.Sources can also include designed reward functions, past behavior, labelled examples, video, audio, and text.Source: §3.1 · pp. 6–7

3.2 Temporality

  • Planned assistance arrives at a set time, whereas interactive advice recurs during operation and can respond to the agent’s state, but requires continuous observation.Interactive feedback can help an agent learn faster than autonomous RL, at the cost of the source continually judging what is relevant.Source: §3.2 · pp. 7–8

3.3 Advice Interpretation

  • Advice interpretation converts inputs such as audio, video, text, or prior behavior into agent-usable formats; transfer learning also requires mapping states between domains.Interpretation may be performed manually or automated in transfer learning; multimodal inputs can be integrated into a state-action pair.Source: §3.3 · pp. 8–9

3.4 Advice Structure

  • Advice structures range from binary feedback and state-action pairs to rules that generalize across states, persist for reuse, and reduce human guidance requirements.Rule-based advice can handle potentially inaccurate or irrelevant information and is reported to improve agent performance.Source: §3.4 · pp. 9–10

3.5 External Model

  • External models retain information for later use or pass it directly; retention suits costly, general, or incremental information, while immediate transfer suits current-step advice or cheap reacquisition.A hybrid can retain contextual affordances while discarding interactive feedback after its current use.Source: §3.5 · p. 10

3.6 Agent Modification

  • Agent modification can shape rewards, policy, learning parameters, or environment; reward shaping must preserve zero-sum rewards to avoid exploiting incentives misaligned with the goal.Environmental changes may be infeasible outside digital or simulated settings, while internal parameter adjustment can be difficult to calibrate.Source: §3.6 · pp. 10–11

3.7 Assisted Agent

  • Without external information, the assisted agent should explore and interact with its environment autonomously like a standard RL agent.The taxonomy treats the agent as the recipient of advice, not as dependent on its presence.Source: §3.7 · p. 11

4 Illustrative Approaches with Components and Links from the Taxonomy

This section applies the ARL taxonomy to illustrative approaches involving multiple information sources, showing how advice and information are represented, processed, combined, and communicated. It covers how planned or interactive assistance shapes decisions and learning, including how agents handle potentially incompatible expert strategies.

4.1 Heuristic Reinforcement Learning

  • A person gives a RoboCup agent two rules before training; when a rule matches the current state, it shapes action selection, otherwise the agent uses its usual decision process [58].The rules are translated into machine-compatible conditions and actions, and the external model can retain the rules and statistics about their use or coverage.Source: §4.1 · pp. 12–13

4.2 Interactive Reinforcement Learning

  • Interactive RL uses human or simulated advice before, during, or after learning, modifying rewards or action selection through reward- or policy-shaping.Advice can assess past actions, recommend actions, or signal rewards; it may be retained by state or used immediately, and the agent can request help when uncertain.Source: §4.2 · pp. 13–14

4.3 Reinforcement Learning from Demonstration

  • RL from demonstration combines reward-based evaluation with expert demonstrations, using demonstrations to initialize a value function or derive a policy that bootstraps autonomous RL.In HAT, the agent records teleoperated state-action transitions, summarizes them as policy rules, then learns independently with the summary as a bias [115].Source: §4.3 · pp. 14–15

• External model:

  • In HAT, demonstrations are encoded as state-action sequences and classified into rules that map states to hypothesized demonstrator actions [115].The generated rules remain fixed in the external model, which can be queried by state.Source: §4.3 · p. 15, • External model · p. 15

• Assisted agent:

  • The HAT agent may query the external model for the demonstrator’s action during decision-making, or otherwise act as a usual RL agent.The proposed action can bias action selection through a Q-value bonus, an added action, or probabilistic execution.Source: • External model · p. 16, • Assisted agent · p. 16

4.4 Transfer Learning

  • Transfer learning typically provides planned assistance before target learning, using knowledge from a source agent; the target can learn to outperform transferred information and may learn when to ignore it.Source: §4.4 · p. 17

4.5 Multiple Information Sources

  • Multiple information sources can broaden expertise but require agents to handle conflicting advice and assess advisor reliability, potentially by state.Sources may offer planned demonstrations or interactive feedback, and heterogeneous advice can be integrated from all, some, or one source at a time.Source: §4.5 · pp. 17–18
  • Advice from multiple sources can be integrated into a single structure using multimodal functions or graphs that link features through causal relations.Possible advice structures include machine rules, state-action pairs, rule systems, values, or models.Source: §4.5 · p. 18
  • Agents may maintain source-specific, combined, or hybrid models; averages source-specific feature weights, while separate models can track advice reliability and avoid difficulties with incompatible sources.Single-model approaches may struggle when sources are fundamentally incompatible.Source: §4.5 · p. 19
  • Existing learning-from-demonstration, transfer-learning, reward-shaping, and inverse-RL modifications can extend to multiple sources by combining models or selecting one best model.The cited approaches combine models with or without weighting, or select a single best model.Source: §4.5 · p. 19
  • For multiple experts with different strategies, Tanwani and Billard learn multiple policies to satisfy expert strategies, including ones not demonstrated, linking the approach to multiobjective RL.The setting is inverse reinforcement learning from demonstrations supplied by multiple experts.Source: §4.5 · p. 19

5 Future Directions and Open Challenges

The section identifies open questions for assisted reinforcement learning (ARL), from unreliable assistance and conflicting sources to communication, explainability, and deployment challenges. Many issues are shared with autonomous RL, but their implications for externally influenced agents remain open.

5.1 Incorrect Assistance

  • Even small amounts of incorrect advice may harm learning; information can be inaccurate, stale, or valid for most states but wrong in critical exceptions.Sources may also provide late advice or actively sabotage performance.Source: §5.1 · pp. 19–20

5.2 Multiple Information Sources

  • Multiple sources can increase environmental knowledge and decision confidence when they agree, but disagreement, source selection, conflict management, and trust remain open questions.Crowdsourcing adds risks from malicious users, anonymity, and uncertainty about information reliability.Source: §5.2 · pp. 20–21

5.3 Explainability

  • When external models or policy modifications make agent choices harder to understand, explanations can help people track decisions and accountability.The authors connect this need to AI ethics and human-computer teaching.Source: §5.3 · p. 21

5.4 Two-Way Communication

  • Most current ARL methods lack free two-way interaction with non-expert advisors; timing and when agents should request assistance remain open design challenges.Requests should balance usefulness against nuisance, learning time, and cost.Source: §5.4 · p. 21

5.5 Other Challenges

  • ARL deployment raises challenges involving agent-specific advice, real-time inference, assistance delays, continuous spaces, safety, partial observability, and competing objectives.Advice may be detrimental to another agent or unsafe; delays can make assistance conflict with autonomous operation, while high-dimensionality and unclear states can make guidance inaccurate. Assistance may also skew subgoal priorities or transfer irrelevant subgoals.Source: §5.5 · pp. 21–22

6 Conclusions

The paper reviews assisted reinforcement learning methods and presents a framework and taxonomy for describing how external information supplements environmental input. It applies the framework across several ARL fields and identifies future research possibilities.

6 Conclusions

  • The review presents an ARL framework for methods that use external information to supplement what agents receive from the environment.The paper describes its contribution as both a state-of-the-art review and a taxonomy offering an additional level of abstraction.Source: §6 · pp. 22–23
  • The taxonomy classifies externally influenced agents through seven features: four processing components and three communication links.The authors note that methods need not use every concept and that simplified models can still fit the framework.Source: §6 · pp. 22–23
  • The framework is applied to heuristic RL, interactive RL, learning from demonstration, transfer learning, and multiple information sources, alongside future research ideas.These fields are analysed and described using the proposed taxonomy.Source: §6 · p. 23
Loading 2007.01544v2…