Source-linked AI summary
Learning to reinforcement learn
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, Matt Botvinick
TL;DR
Deep RL often demands massive training data, creating a need for rapid adaptation to new tasks. The paper trains recurrent networks with deep RL so their dynamics implement a separate, task-distribution-shaped RL procedure, which it examines across seven proof-of-concept experiments. The results include one-shot task-structure learning and robust performance in several task settings, while the authors note that model-based-looking behavior may arise from model-free representations.
Problem
Deep RL systems can achieve superhuman performance but typically require massive training data, motivating rapid adaptation to new tasks.
Method
Deep meta-RL trains a recurrent neural network with one RL algorithm so its dynamics implement a separate RL procedure adapted to interrelated task distributions.
Results
Across seven proof-of-concept experiments, the approach showed task-structure learning, including 43% of stacked-LSTM seeds reaching ceiling versus 26% for single-LSTM seeds.
Takeaways & Limitations
The learned recurrent procedure can exploit training-domain structure and support rapid adaptation to new task instances.
Takeaways & Limitations
Model-based-looking behavior in the two-step task may instead arise from a model-free system with an appropriate state representation.
Abstract
from arXiv · showhide
In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.
1 INTRODUCTION
Deep RL achieves strong performance but typically requires massive training data, motivating methods that adapt rapidly to new tasks. The paper proposes deep meta-RL and examines it through seven proof-of-concept experiments.
- Deep RL reaches human- and often superhuman-level performance across expanding task domains but typically requires massive training data.
- Deep meta-RL trains a recurrent neural network whose dynamics implement a separate RL procedure tailored to the training domain.
- The paper extends recurrent-network meta-learning from supervised learning to reinforcement learning.
- Seven proof-of-concept experiments characterize key aspects of deep meta-RL, followed by discussion of scaling challenges and neuroscience implications.
2 METHODS
The method trains recurrent networks across related tasks so their internal dynamics learn a task-distribution-dependent RL algorithm. After training, frozen weights support adaptation through changing activations and hidden states.
- Meta-learning uses related tasks sharing regularities so an agent improves more rapidly on new tasks.
- Recurrent-network meta-learning receives prior-step target information, allowing its dynamics to learn within-task behavior from task-family structure.
- Unlike supervised recurrent meta-learning, deep meta-RL applies the approach to interrelated reinforcement-learning problems.
- Deep meta-RL uses a recurrent network trained with standard deep RL so its dynamics implement a separate, full-fledged RL algorithm.
- The learned procedure can adapt its policy updates and exploration to the statistics of the multi-task environment.
- During evaluation, weights are frozen while activations and recurrent hidden states change in response to environmental inputs.
3 EXPERIMENTS
The experiments evaluate recurrent meta-RL across bandit and Markov decision problems using a common LSTM-centered architecture trained with actor-critic methods. Variants incorporate task-specific inputs, reward and action history, and visual encoders.
- The study reports six experiments plus a seventh related experiment, covering four bandit tasks and two Markov decision problems.
- All experiments center on an LSTM feeding a soft-max discrete-action policy, with minor architectural variations across tasks.
- Reward and the previous action are generally supplied as recurrent-network inputs to support within-task learning.
- Advantage Actor-Critic trains the reinforcement-learning systems, using entropy regularization and combined policy and value-function losses.
- The architecture includes recurrent, convolutional-LSTM, and stacked-LSTM variants for experiments spanning bandits, abstract task structure, and visual tasks.
3.1 BANDIT PROBLEMS
The bandit experiments show that recurrent agents can learn prior-dependent bandit algorithms that exploit structure in their training distributions, generalize to related distributions, and adapt behavior to information and volatility.
- Meta-RL learns a prior-dependent bandit algorithm that performs well on environments drawn from its training distribution and can generalize to related distributions.
- The learned policy is evaluated with cumulative regret, measuring expected reward loss from selecting sub-optimal arms, or with the number of sub-optimal pulls.
- Bandits with independent arms: On independent Bernoulli arms, meta-RL outperformed Thompson sampling and UCB but performed worse than Gittins; removing reward input reduced performance to chance.
- Bandits with dependent arms: In correlated-arm bandits with p1 = 1 − p2, structured-task training produced prior knowledge that performed comparably to Gittins on structured tests and exceeded independent-arm training.
- Bandits with dependent arms: Training on correlated arms hurt performance on independent tests, consistent with learned policies relying on reward structure specific to the training distribution.
- Bandits with dependent arms: The learned procedure also sampled an informative low-paying arm once to gain information, then exploited the identified high-value arm for long-run returns.
- Restless bandits: In volatile bandits, meta-RL achieved lower regret than Thompson sampling, UCB, and fixed-rate Rescorla-Wagner learning, while its effective learning rate increased with volatility.
3.2 MARKOV DECISION PROBLEMS
These experiments test whether deep meta-RL can adapt to structured MDPs, including model-based decision patterns, abstract one-shot learning, and navigation with sparse rewards. Recurrent agents exploit task structure after training with standard deep RL.
- THE “TWO-STEP TASK”: The two-step task dissociates model-free value caching from model-based look-ahead planning through stay probabilities after common or rare transitions.The task uses probabilistic transitions and differently rewarded second-stage states.
- THE “TWO-STEP TASK”: Model-free A2C training produced choices conventionally interpreted as model-based control in the modified two-step task.With reward withheld from the network input, performance fell to chance.
- LEARNING ABSTRACT TASK STRUCTURE: The final experiment combined rich visual inputs, longer horizons, sparse rewards, and an abstract task structure requiring inference of object roles.This setting was designed to probe meta-RL scalability.
- LEARNING ABSTRACT TASK STRUCTURE: In the visual Harlow task, stacked LSTMs were more robust than single LSTMs: 43% versus 26% of seeds reached ceiling performance.The comparison used the best hyperparameter set.
- LEARNING ABSTRACT TASK STRUCTURE: After one trial, agents selected the rewarded image from the second trial onward despite changing image positions, demonstrating one-shot learning of task roles.The agent also learned the fixation behavior required at trial onset.
- ONE-SHOT NAVIGATION: In navigation, a stacked LSTM learned goal location after exploration: first visits took ~100 timesteps, versus ~30 timesteps for subsequent visits.The environment layout stayed fixed while the goal location changed randomly each episode; feedforward networks did not solve the task.
4 RELATED WORK
Deep meta-RL builds on recurrent-network meta-learning while differing from earlier meta-RL and related optimization work. Its distinctive emphasis is learning efficient procedures for structured task distributions.
- RELATED WORK: Deep meta-RL extends recurrent-network meta-learning from supervised learning to reinforcement learning.In this framework, learning within a task is implemented by recurrent dynamics rather than by the procedure that trained the weights.
- RELATED WORK: Earlier work introduced the term meta-RL but did not use a neural-network implementation, whereas this paper trains recurrent neural networks with deep RL.The paper also relates to neural methods for learning optimization procedures.
- RELATED WORK: Related navigation studies train recurrent networks on environments whose task structure varies across episodes, including goal locations or maze configurations.The paper’s navigation experiment is presented as one example of this relationship.
- RELATED WORK: Duan et al. likewise learned a second, faster-timescale RL procedure through deep RL training, but focused mainly on relatively unstructured task distributions.This paper instead emphasizes structured distributions where learned biases can exploit regularities for efficiency.
5 CONCLUSION
Deep meta-RL trains recurrent networks whose dynamics implement a separate, learned RL procedure tuned to shared structure across related tasks. This can produce domain-appropriate biases, including covariance-sensitive learning, one-shot learning, and behavior resembling model-based control.
- Deep meta-RL combines deep RL training, interrelated tasks, and previous action-reward inputs so recurrent dynamics learn a separate RL procedure.The learned procedure can differ from the algorithm used to train the network weights.
- The learned RL algorithm is tuned to shared task structure, building in domain-appropriate biases that can improve efficiency over a general-purpose algorithm.
- Dependent-bandit experiments showed that the system exploited covariance structure, while Harlow’s task produced one-shot learning with complex novel stimuli.
- A model-free training algorithm produced behavior emulating model-based control, although the authors suspect the network may instead use an appropriate state representation.
- Deep meta-RL may have neuroscience implications, including an integrative framework for the roles of dopamine and prefrontal cortex in biological reinforcement learning.