RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Yan Duan; John Schulman; Xi Chen; Peter L. Bartlett; Ilya Sutskever; Pieter Abbeel

Source-linked AI summary

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel

arXiv:1611.02779v2Nov 9, 2016cs.AIcs.LGcs.NEstat.ML

TL;DR

Deep RL often needs many trials, while humans and animals can learn tasks quickly, motivating methods that use prior knowledge efficiently. RL² learns a recurrent agent as a reinforcement-learning algorithm across MDPs, and it performs comparably with theoretically justified methods on small problems while scaling to vision-based navigation. The paper also identifies outer-loop optimization and episodic structure as important improvement opportunities.

Problem
Deep RL can require huge numbers of trials, whereas humans and animals learn new tasks in few trials; existing prior-informed methods face computational or environmental constraints.
Method
RL² represents the fast RL algorithm as an RNN whose weights are learned by standard RL over a distribution of MDPs, preserving hidden state across episodes.
Results
RL² performs comparably with theoretically justified algorithms on small-scale problems and demonstrates potential to scale to high-dimensional tasks.
Takeaways & Limitations
Learning an RL algorithm end-to-end can produce an agent that adapts to new tasks and can operate on high-dimensional problems.
Takeaways & Limitations
The outer-loop reinforcement-learning algorithm is an immediate bottleneck, and better architectures may be needed for extremely long horizons.

Abstract

from arXiv · show

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL$^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL$^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

1 INTRODUCTION

Deep RL can learn sophisticated behaviors but often requires far more experience than humans and animals. RL² instead learns a recurrent agent that adapts across tasks, achieving near-reference performance on classical problems and scaling to vision-based navigation.

Motivation: Deep RL has achieved sophisticated behaviors, but Atari results can require tens of thousands of episodes per game.The paper contrasts this sample complexity with humans and animals learning new tasks in very few trials.
Motivation: Bayesian reinforcement learning incorporates prior knowledge, but exact Bayesian updates are intractable beyond simple cases.Practical alternatives can reduce experience requirements while imposing environmental assumptions or becoming computationally intractable in high-dimensional settings.
Approach: RL² learns the agent’s learning process by optimizing an RNN across a distribution of MDPs using standard reinforcement learning.The RNN receives observations, actions, rewards, and termination flags, while its internal state persists across episodes.
Evaluation: RL² performs comparably with theoretically justified algorithms on multi-armed bandits and tabular MDPs, and scales to vision-based navigation.The evaluation covers classical small-scale problems and a high-dimensional ViZDoom navigation task.

2 METHOD

RL² casts learning an RL algorithm as a reinforcement-learning problem over repeated episodes on sampled MDPs. Its recurrent policy preserves hidden state within each trial, enabling adaptation from interaction while standard RL optimizes trial-level return.

2.1 PRELIMINARIES: An MDP is defined by states, actions, transition probabilities, rewards, an initial-state distribution, a discount factor, and a finite horizon.The policy maximizes expected discounted trajectory return.
2.2 FORMULATION: RL² samples an MDP from a distribution and treats its allowed sequence of episodes as one trial.Only sampling from the MDP distribution is required.
2.2 FORMULATION: Each episode supplies the policy with the next state, action, reward, and termination flag, while the hidden state carries across episodes but resets between trials.This interaction structure is illustrated in Figure 1.
2.2 FORMULATION: Maximizing discounted reward over a trial forces the agent to integrate interaction history and continually adapt to the current MDP.The paper relates this objective to minimizing cumulative pseudo-regret.
2.2 FORMULATION: The formulation also applies conceptually to partially observed settings, including the visual navigation experiment.In that setting, observations replace states and the inner problems are POMDPs.
2.3 POLICY REPRESENTATION: The policy is represented as a recurrent neural network that receives embedded state, action, reward, and termination inputs.The implementation uses GRUs to reduce difficulties from vanishing and exploding gradients.
2.3 POLICY REPRESENTATION: Alternative architectures that explicitly reset part of the hidden state each episode did not improve performance over the simple architecture.The comparison was performed on the sampled-MDP episodic setting.
2.4 OPTIMIZATION: Standard off-the-shelf reinforcement learning algorithms optimize the recurrent policy, with a GRU-based recurrent baseline used to reduce gradient-estimation variance.The outer-loop optimizer is a first-order implementation of TRPO.

3 EVALUATION

RL2 is evaluated on bandits, tabular MDPs, and visual navigation to test whether learned policies can match specialized algorithms and scale to high-dimensional tasks. Results show near-reference performance on bandits, strong performance for short tabular-MDP trials, and cross-episode information use in navigation, with limitations in the hardest bandit setting and imperfect memory reuse.

Evaluation goals: RL2 is evaluated on multi-armed bandits, tabular MDPs, and vision-based navigation.The evaluation compares learned behavior with established strategies on classical problems and tests scaling to high-dimensional inputs.
3.1 MULTI-ARMED BANDITS: RL2 achieves performance almost as good as human-designed reference methods on multi-armed bandits.The references include random, Gittins index, UCB1, Thompson sampling, optimistic Thompson sampling, epsilon-greedy, and greedy strategies.
3.1 MULTI-ARMED BANDITS: A noticeable gap remains between Gittins index and RL2 with 50 arms and 500 episodes, while supervised training matches Gittins index performance.This result suggests the bottleneck lies in the slow reinforcement-learning optimization rather than the policy architecture.
3.2 TABULAR MDPS: In tabular MDPs, RL2 outperforms existing methods for fewer episodes but loses its advantage as the episode count increases.The authors attribute the short-horizon advantage to earlier exploitation when limited samples make estimating the full dynamics difficult.
3.3 VISUAL NAVIGATION: Visual navigation shows shorter trajectories between the first two episodes in both maze sizes, indicating use of information from past episodes.Performance is maintained in further episodes, but larger mazes show a small success-rate drop and fewer improved trajectories.
3.3 VISUAL NAVIGATION: Even in small mazes, RL2 sometimes forgets the target location and continues exploring during the second episode.The intended behavior is to remember the target and act optimally after locating it.

4 RELATED WORK

RL² builds on prior work in meta-learning, Bayesian reinforcement learning, transfer, and POMDP formulations for exploiting prior experience and handling unknown environments.

Meta-learning and Bayesian reinforcement learning: Earlier studies used hyperparameter tuning and hierarchical Bayesian methods to incorporate prior experience into reinforcement learning.Examples include automatic tuning of learning rates and temperatures, and posterior-based optimistic Thompson sampling over dynamics models.
Transfer and one-shot learning: Model-based approaches reduced sample complexity on related tasks by learning neural-network priors for unknown dynamics.Fu et al. used samples from previous tasks to enable one-shot learning on new, related tasks.
End-to-end meta-learning: Meta-learning has also been formulated as an optimization problem that can be trained end-to-end with gradient descent.This line of work includes one-shot learning for object classification and related neural meta-learning methods.
POMDP formulation: RL² differs from directly parametrized-policy approaches because its recurrent neural network is both the meta-learner and the resulting policy.The formulation also casts an unknown underlying MDP as a POMDP solved in the outer loop, connecting it to dual control and mixed-observability MDPs.

5 DISCUSSION

RL2 learns reinforcement-learning algorithms end-to-end, storing the fast algorithm’s state in RNN activations while learning the weights with a slow RL algorithm. It performs comparably to theoretically optimal algorithms in small settings but has identified scalability bottlenecks.

RL2 encodes the fast RL algorithm in RNN activations and learns its weights using a general-purpose slow RL algorithm.
RL2 demonstrates competence comparable with theoretically optimal algorithms in small-scale settings and potential to scale to high-dimensional tasks.
The outer-loop RL algorithm is an immediate bottleneck, while extremely long horizons may require better policy architectures.

A DETAILED EXPERIMENT SETUP

The experiments use placeholder values for unavailable initial inputs, standardized neural-network initialization and normalization, and TensorFlow with rllab and TabulaRL implementations.

At t = 0, placeholder embeddings and zeros substitute for unavailable previous actions, rewards, and termination flags.Action 0 supplies the placeholder embedding; rewards and termination flags use 0.
The networks use rectified linear hidden activations, weight normalization, orthogonal hidden-to-hidden initialization, Xavier initialization elsewhere, and zero biases.
All experiments use TensorFlow and rllab, while classic algorithms come from the TabulaRL package.

A.1 MULTI-ARMED BANDITS

For multi-armed bandits, the setup uses a constant state placeholder and one-hot action embeddings; TRPO hyperparameters are listed separately.

Because the environment is stateless, a constant embedding 0 replaces the state input and actions use one-hot embeddings.
Table 1 lists the TRPO hyperparameters for multi-armed bandits.

A.2 TABULAR MDPS

For tabular MDPs, states and actions are represented with separate one-hot embeddings that are concatenated; TRPO hyperparameters are listed in Table 2.

Tabular MDP states and actions use separate one-hot embeddings, which are then concatenated.
Table 2 lists the TRPO hyperparameters for tabular MDPs.

A.3 VISUAL NAVIGATION

The visual navigation setup preprocesses RGB images, combines visual and action embeddings, and uses a shared network for the policy and baseline. TRPO hyperparameters are listed in Table 3.

Images are resized to 40 × 30 pixels with RGB channels preserved and recentered to [−1, 1].
Two convolutional layers with 16 filters each process the image, while a learned 256-dimensional action embedding is concatenated with the flattened visual features.
The concatenated joint vector is fed into a fully connected layer with 256 hidden units.
Sharing one neural network between the policy and baseline improved baseline-training stability and policy end performance, possibly through regularization and better learned features.
Table 3 reports the hyperparameters used for TRPO in visual navigation.