Source-linked AI summary
Been There, Done That: Meta-Learning with Episodic Recall
Samuel Ritter, Jane X. Wang, Zeb Kurth-Nelson, Siddhant M. Jayakumar, Charles Blundell, Razvan Pascanu, Matthew Botvinick
TL;DR
Meta-learning agents learn new tasks efficiently, but recurring tasks require remembering prior solutions rather than exploring again. This paper formalizes open-ended task sequences with recurrence and adds episodic memory to an LSTM-based meta-learning agent. Across five experiments, the agents demonstrated several forms of recall, memory retrieval, and policy reuse.
Problem
Meta-learning emphasizes efficient learning on new tasks, while how to recognize recurring tasks and retrieve prior learning remains a separate challenge in open-ended environments.
Source: §1 · p. 1Method
Results
Across five experiments, agents recalled discovered policies, retrieved memories from category examples, handled compositional tasks, reinstated memories in multi-state MDPs, and discovered a human episodic learning algorithm in a neuroscience-inspired task.
Source: §5 · p. 8Takeaways & Limitations
The work constitutes a first step toward deep RL agents that exploit both structure and repetition in their environments.
Source: §5 · p. 8Experiments use readily identifiable contexts and clear episode boundaries, whereas real-world agents may need to infer contexts over time and decide when to save or forget memories.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Meta-learning agents excel at rapidly learning new tasks from open-ended task distributions; yet, they forget what they learn about each task as soon as the next begins. When tasks reoccur - as they do in natural environments - metalearning agents must explore again instead of immediately exploiting previously discovered solutions. We propose a formalism for generating open-ended yet repetitious environments, then develop a meta-learning architecture for solving these environments. This architecture melds the standard LSTM working memory with a differentiable neural episodic memory. We explore the capabilities of agents with this episodic LSTM in five meta-learning environments with reoccurring tasks, ranging from bandits to navigation and stochastic sequential decision problems.
1. Introduction
Meta-learning supports efficient learning across open-ended task sequences, but recurring tasks raise a separate challenge: agents should recognize them and retrieve what they learned before. The paper frames this as a gap between meta-learning’s focus on novel tasks and continual-learning work that generally studies smaller task sets.
1. Introduction
- Meta-learning addresses how agents cope with open-ended environments that present an unbounded series of tasks.It concerns improving an agent’s own learning efficiency and effectiveness through experience.Source: §1 · p. 1
- Meta-learning research typically emphasizes efficient learning on new tasks, leaving the response to a previously mastered task’s recurrence as a second, equally important problem.Ideally, the learner recognizes the task, retrieves prior learning, and picks up where it left off.Source: §1 · p. 1
- Continual-learning work considers recurring tasks but generally avoids the open-ended settings and few-shot learning demands central to meta-learning.Such work commonly assumes a small set of tasks.Source: §1 · p. 1
2. Problem Formulation
The formulation generates sequences of tasks by sampling MDPs together with contexts from stochastic task processes. These processes can control recurrence patterns, while the experiments chiefly use uniform sampling from a bag containing duplicate tasks.
2. Problem Formulation
- Tasks are MDP-context pairs sampled from stochastic task processes, extending repeated sampling from a task distribution to sequences with identifiable recurrence.The process has base distribution D and parameters θ; the paper denotes each pair (m_n, c_n) as task t_n.Source: §2 · p. 1
- A Blackwell-MacQueen urn scheme sets recurrence through a rich-get-richer process in which each task occurrence increases its chance of recurring.It samples a new task with probability α/(α+n); otherwise it draws an urn task and adds both copies back, producing Zipf-like distributions.Source: §2 · p. 2
- The experiments primarily use a hypergeometric process that samples uniformly without replacement from a bag containing duplicate tasks.The intended agent should both learn faster from shared structure and avoid renewed exploration when a task recurs.Source: §2 · p. 2
3. Agent Architecture
The episodic LSTM adds a differentiable long-term memory to an L2RL agent, storing task-related cell states so they can be retrieved when a context recurs. A reinstatement gate incorporates the retrieved state directly into working memory alongside the usual input and prior-state contributions.
3. Agent Architecture
- The episodic LSTM stores LSTM cell states with contextual cues and reinstates a stored state when a similar cue recurs, addressing L2RL’s task-switch forgetting.In L2RL, the cell state retains results of exploration within a task but is reset for the next task.Source: §3 · p. 2
- A differentiable neural dictionary stores context embeddings as keys and LSTM cell states as values, retrieving prior exploration results by context similarity.The DND uses k-nearest-neighbor search over keys.Source: §3 · pp. 2–3
- Retrieved states are added directly to the current cell state rather than passed through learned transformations as extra network inputs.The paper motivates this choice because retrieved and current states share dimensionality and semantics, while learned transformations have proved difficult in RL.Source: §3 · p. 3
- A reinstatement gate coordinates the retrieved state with the input and forget-gated contributions to working memory.The retrieved episodic state contributes through a new term in the LSTM cell update.Source: §3 · p. 3
4. Experiments
Across five experiments, epL2RL is tested on recurring tasks ranging from bandits to navigation and sequential decisions. The results show that agents can use episodic memory to benefit from previous exposures, including when task cues vary or task components recombine.
4. Experiments
- Across five experiments, the evaluation spans recurring bandit tasks, water-maze navigation, and an episodic two-step decision task.The bandit experiments progress from whole-task recurrence to varying Omniglot contexts and recombining task components.Source: §4 · p. 3
4.1. Experiment 1: Barcode Bandits
- In barcode bandits, epL2RL widely outperformed L2RL and L2RL+Context, which supplied barcodes as LSTM inputs.Feeding retrieved DND values as additional LSTM inputs instead of using the r-gate reduced performance to random chance.Source: §4.1 · pp. 4–5
4.2. Experiment 2: Omniglot Bandits
- In Omniglot bandits, epL2RL’s regret decreased with successive exposures, and it outperformed Gittins indices, UCB, and Thompson sampling after a few exposures.Unlike L2RL without episodic memory, it improved as exposures increased; weights were frozen during evaluation, supporting recall of prior exploration.Source: §4.2 · p. 5
4.3. Experiment 3: Compositional Bandits
- In compositional bandits, epL2RL increased reward over L2RL+Context and reduced regret on successive exposures, despite contexts reappearing in different arm positions.The evaluation weights were frozen, so the regret decrease was not attributable to gradient-based learning.Source: §4.3 · p. 6
4.4. Experiment 4: Contextual Water Maze
- In contextual water mazes, epL2RL outperformed L2RL and the maximum average reward achievable without episodic memory; after an initial exposure, it reached goals in fewer steps.With weights frozen, second-exposure trajectories navigated directly to the goal by one of the shortest paths.Source: §4.4 · p. 6
4.5. Experiment 5: Episodic Two-Step Task
- epL2RL learned to use episodic model-based control alongside incremental strategies and earned more reward than L2RL on the episodic two-step task.The fitted model showed IMF and IMB use in uncued episodes, and IMF, IMB, and EMB use in cued episodes.Source: §4.5 · p. 7
4.5.1. ANALYSIS OF THE R-GATES
- In the r-gate analysis, gates were more open on cued episodes with correct than incorrect actions, but the difference was small and the authors describe the evidence as preliminary.Mean gate values were 0.365 for correct and 0.358 for incorrect cued actions; the authors note that other processes may govern the interplay of working memory, retrieved activations, and input.Source: §4.5.1 · pp. 7–8
5. Discussion
Across five experiments, the agents recalled and reinstated task knowledge in several settings, including navigation and a neuroscience-inspired task. The discussion frames this work as an initial step toward deep RL agents that exploit structure and repetition, while identifying practical challenges for real-world use.
5. Discussion
- Across five experiments, agents recalled policies, retrieved memories from category examples, handled compositional tasks, reinstated memories in multi-state MDPs, and discovered a human-like episodic learning algorithm.The authors describe the study as a first step toward deep RL agents that exploit structure and repetition.Source: §5 · p. 8
- Real-world contexts may lack clear labels or pretrained embeddings and may share an input channel with other signals, making learned query keys a critical next step.The authors suggest learning query keys through gradients passed through retrieval or auxiliary training over DND contents.Source: §5 · p. 8
- DND contents could support auxiliary contrastive training at relatively low marginal computational cost, though defining a neighborhood function remains a challenge.The authors also identify passing gradients through retrieval as a future direction for epL2RL.Source: §5 · p. 8
- When context is only partially observable within a timestep, agents may need to aggregate information over time to form an effective query embedding.The authors also note that real-world event boundaries are less clear than the experimental episode boundaries, leaving agents to decide when to save or forget memories.Source: §5 · p. 8
- Reinstatement-based episodic memory may also be useful in other RL settings and sequence-learning problems, which the authors leave for future work.Source: §5 · p. 8