The Barbados 2018 List of Open Issues in Continual Learning

Setting

• RL agents receive a reward signal defining success, but worlds may exceed their capacity to explore, observations their capacity to store, and reward may offer rare signs of progress.Source: Setting · p. 1
• Dynamics and rewards may change, while the agent faces one uninterrupted life and choices that can irreversibly prune possible futures.Source: Setting · p. 1
• The authors hypothesize that these conditions require agents to flexibly learn to predict and control their data stream, forming useful knowledge continually and constructively.Source: Setting · p. 1

What is Continual Learning?

• The central trade-off is between acquiring knowledge for future use and acting to maximize reward in the present.Source: What is Continual Learning? · pp. 1–2
• A vast stationary world can appear non-stationary to an agent with limited resources.Source: What is Continual Learning? · p. 1
• The learning-versus-reward balance may need to be chosen in advance, including to avoid converging to stationary behavior when the world changes or better rewards may exist.Source: What is Continual Learning? · p. 2
• The report calls this class of problems continual learning, describes it as closely related to learning in humans and animals, and says it is roughly synonymous with lifelong learning.Source: What is Continual Learning? · p. 2

Open Issues

• The report presents a list of issues it considers critical for continual learning.Source: Open Issues · p. 2
• It emphasizes issues that are particularly challenging for continual learning, rather than challenges shared across reinforcement learning.Source: Open Issues · p. 2
• Some of the listed issues may also matter outside continual learning or reinforcement learning.Source: Open Issues · p. 2

1. Evaluating continual learning

• Evaluation is framed as instrumental to rapid progress, clear communication, reproducibility, and comparability of methods.Source: §1 · p. 2

1.1. Domains

• Domains should expose continual-learning challenges while enabling measurable progress and meaningful algorithm comparisons, spanning illustrative problems and vast complex worlds.Source: §1.1 · p. 2

1.2. Metrics

• Reward over a lifetime and average reward per step may not suffice; auxiliary probe-question metrics and qualitative behavior analysis could enrich evaluation.Source: §1.2 · p. 2

2. Designing continual learning agents

• Agents must integrate solutions to interrelated challenges to learn to predict and control their data stream.The report expects these challenges are unlikely to be solved entirely in isolation.Source: §2 · p. 2

2.1. Choosing what to learn about (discovery)

• Discovery requires choosing predictive questions that let agents autonomously verify answer quality and represent knowledge relevant to behavior.Determining which questions to ask remains a challenge.Source: §2.1 · p. 2

2.1.1. Representing questions

• General value functions (GVFs) represent agent-state-dependent predictions conditioned on a policy, accumulated signal, and temporal horizon.Agents can maintain many GVFs in parallel, each attending to different signals and horizons.Source: §2.1.1 · pp. 2–3

2.1.2. Generating questions

• Questions can be generated bottom-up from signals or top-down from a need for answers, with one-step transition and reward models as GVF special cases.The report identifies a spectrum between these approaches.Source: §2.1.2 · p. 3

2.1.3. Assessing questions

• Question utility needs criteria beyond the agent’s reward, since a new question may help answer existing questions.Source: §2.1.3 · p. 3

2.2. Gathering the experience needed to learn (exploration)

• Because agents partly control their data stream, exploration must balance effectiveness, novelty, survival, and experience quality under risk.Reward alone may not guide these decisions adequately, and exhaustive exploration may be unsuitable in worlds too large to cover.Source: §2.2 · p. 3

2.2.1. Drives

• Drives can be intrinsic signals or inductive biases that encourage useful experience, from novelty seeking to learning progress and uncertainty reduction.Examples also include controllability and empowerment, and uncertainty reduction in answers.Source: §2.2.1 · p. 3

2.2.2. Building exploration skills

• Agents may need to develop exploration skills, including commitment or persistence and consistent or repeated behavior.Source: §2.2.2 · p. 3

2.2.3. Survival and risk-awareness

• A single-life agent must trade off drives and skills against extrinsic reward while accounting for risk and sensitivity to death.Risk handling may require estimating more than mean outcomes [2].Source: §2.2.3 · p. 3

2.3. Learning in preparation for future questions (knowledge)

• Competent action in a non-stationary world depends on generalizing effectively from past experience to the present.Source: §2.3 · p. 3

2.3.1. Representing many answers

• Answer representations range from separate non-parametric, non-interfering solutions to parametric architectures that generalize across questions, with modular approaches between them.Source: §2.3.1 · p. 3

2.3.2. Learning many answers

• Off-policy learning can answer questions conditioned on policies other than current behavior, but efficient off-policy learning remains open despite many existing methods.Source: §2.3.2 · p. 4

2.3.3. Learning from few examples

• Because only a tiny fraction of states may ever be encountered, continual learning treats data as precious and calls for using it effectively.Source: §2.3.3 · p. 4

2.3.4. Appropriate generalization

• Generalization must span states, questions, and time, while non-stationarity makes it uncertain whether change is permanent or periodic.Capacity-limited agents must balance learning from new experience against forgetting or overwriting old knowledge—the stability-plasticity dilemma.Source: §2.3.4 · p. 4

2.3.5. Balancing learning effort and inference effort

• Limited memory and compute constrain both learning and action inference, creating a trade-off between them.Flexible memory can cache decision-making computation, such as action-value functions in place of replanning each step.Source: §2.3.5 · p. 4

2.3.6. Trading off multiple objectives

• Continual learning may require preserving multiple competing objectives rather than maximizing one at the expense of others.Whether fixed weighting or an explicit multi-objective formulation is more effective remains unclear.Source: §2.3.6 · p. 4

2.4. How to use knowledge

• Using learned knowledge about non-reward quantities to improve reward-seeking effectiveness can be non-trivial.Source: §2.4 · p. 4

2.4.1. Using knowledge to improve learning

• Shared representations across questions can support generalization and reward exploitation, and answers to some questions can serve as features for learning others [5].Source: §2.4.1 · p. 4

2.4.2. Using knowledge to improve behaviour

• Knowledge about controlling or optimizing environmental signals can support more directed behavior than low-level motor dithering.Such behavior can help obtain interesting experience or reward.Source: §2.4.2 · p. 4

2.4.3. Using knowledge to infer more knowledge

• Full world models could enable optimal planning without further exploration, but building models and planning with them remains open.One-step raw-observation models may waste resources on irrelevant signals and accumulate errors over long horizons; planning methods may need to handle inaccurate, partial, or abstract models.Source: §2.4.3 · p. 4

• The list aims to help researchers study continual learning by building on existing reinforcement-learning ideas and algorithms.Source: Conclusion · p. 4