Source-linked AI summary
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov
TL;DR
The paper addresses the challenge of learning across multiple environments and transferring knowledge to new tasks. Actor-Mimic trains a single policy network by mimicking several expert DQNs, then uses its pretrained representations to initialize learning on related target tasks. The resulting network reaches near-expert performance on many source games and can significantly improve learning speed on target games, although transfer is limited when tasks differ substantially.
Problem
Existing DQN training learns each game separately despite similarities between games, while DRL training is computationally intensive; multitask and transfer learning could share representations and reduce training time.
Method
Actor-Mimic uses model-compression techniques to train one policy network by matching the policies of game-specific expert DQNs, then initializes target-task DQNs with its pretrained weights.
Results
Actor-Mimic reaches expert performance on many games simultaneously and multitask pretraining significantly improves learning speed on a set of target tasks.
Takeaways & Limitations
Source-task features can generalize to new target tasks when source and target tasks have sufficient similarity.
Takeaways & Limitations
Transfer is not uniformly beneficial: it has little positive effect for Gopher and Robotank and can slow learning for Robotank, which differs substantially from the source tasks.
Abstract
from arXiv · showhide
The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed "Actor-Mimic", exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.
1 INTRODUCTION
Deep reinforcement learning learns useful representations directly from raw, high-dimensional inputs, but conventional DQN training handles only one game at a time. Actor-Mimic addresses this limitation by training one policy across multiple games and using its learned representations for transfer to new tasks.
- Deep reinforcement learning combines reinforcement learning with deep neural-network function approximators for challenging high-dimensional tasks.It can learn salient descriptions of raw state inputs, reducing the need for manually engineered features.
- DQN uses the same architecture and hyperparameters across games, but each network learns only one game at a time despite inter-game similarities.Pong and Breakout share a moving-ball-and-paddle structure that could support knowledge sharing.
- A multitask network could exploit similarities among games to learn a compact shared state representation and transfer knowledge to new tasks.Training on source tasks may speed learning in target tasks, while reducing training time is practically valuable because DRL training is computationally intensive.
- Actor-Mimic trains a single multitask policy network using guidance from several game-specific expert networks, then evaluates transfer in the Atari Learning Environment.The method applies model-compression techniques to multitask and transfer learning for DRL agents.
2 BACKGROUND: DEEP REINFORCEMENT LEARNING
Deep reinforcement learning formulates control as decision-making in a Markov Decision Process and estimates action values to select behavior. Because Atari state spaces are too large for tabular methods, DQN uses neural-network approximations supported by replay memory.
- A Markov Decision Process is defined by states, actions, transition probabilities, rewards, and a discount factor.An agent’s behavior is represented by a policy that specifies action probabilities for each state.
- The Q-function represents expected future discounted reward from taking an action in a state and then following a policy.The optimal Q-function satisfies a Bellman equation, and an optimal policy chooses the action with the highest Q-value.
- Q-learning iteratively updates action values toward the optimal Q-function, but tabular storage is intractable for the Atari Learning Environment’s large state space.Deep Q-Networks address this setting by using neural-network function approximation.
- Replay memory stores previous transition tuples and samples them uniformly during training.This reduces correlations between adjacent states and can substantially improve training stability in some games.
3 ACTOR-MIMIC
Actor-Mimic trains one multitask policy from several expert networks by matching their actions and internal features, then reuses the learned representation to initialize transfer to related tasks.
- Policy regression: Actor-Mimic trains a single multitask network using guidance from expert DQN networks specialized in different source games.The method uses model-compression techniques to learn behavior across source tasks in one network.
- Policy regression: Softmax-transformed expert Q-values produce bounded policy outputs that reduce differences in value-function scales and emphasize the expert’s selected action.The expert DQN is converted into a policy through a Boltzmann distribution over Q-value outputs.
- Policy regression: The multitask network is trained with cross-entropy against expert policies, providing a supervised target instead of a recursively self-dependent Q-learning target.Training trajectories can be sampled from either the expert or the learning Actor-Mimic network; empirically, sampling from Actor-Mimic performed best.
- Feature regression: Feature regression trains mappings from Actor-Mimic hidden activations to each expert’s features, with error back-propagated through Actor-Mimic.This encourages the multitask representation to contain information predictive of expert representations, even when feature dimensions differ.
- Actor-Mimic objective: The combined Actor-Mimic objective adds policy and feature regression losses, with β controlling their relative weighting.Policy regression guides how the network acts, while feature regression guides the representations associated with those actions.
- Transfer: For transfer, the final softmax layer is removed and the pretrained Actor-Mimic weights initialize a DQN trained on a related target task.The initialization supplies features learned to define policies in related source tasks, which may remain useful after fine-tuning.
4 CONVERGENCE PROPERTIES OF ACTOR-MIMIC
The analysis establishes convergence of Actor-Mimic under stated MDP and policy-regularity assumptions, then derives performance guarantees relative to an expert. Sampling from the learned Actor-Mimic policy yields linear cost growth, whereas expert-only sampling yields quadratic growth.
- 4.1 STOCHASTIC STATIONARY POLICY: The convergence analysis represents states with compact K-dimensional features and models Q-values linearly before applying a softmax policy.The resulting policy is defined from the linear approximator’s action scores.
- 4.1 STOCHASTIC STATIONARY POLICY: Under a fixed policy, the policy-regression objective is convex with a unique stationary point, enabling almost-sure convergence of stochastic gradient updates.Compact input features provide a bounded Lipschitz constant for the objective.
- 4.2 STOCHASTIC ADAPTIVE POLICY: With irreducible and aperiodic MDPs and a Lipschitz-continuous policy operator, the iterative sequence of policies and parameters converges almost surely to a unique solution.These conditions apply to every policy induced by the operator Γ.
- 4.3 PERFORMANCE GUARANTEE: The convergence result places Actor-Mimic among no-regret online-learning algorithms and supports a performance bound relative to the guiding expert.The cited analysis provides a guarantee on how the Actor-Mimic model performs with respect to that expert.
- 4.3 PERFORMANCE GUARANTEE: Linear cost growth follows when trajectories sample the Actor-Mimic action output, while expert-only sampling produces quadratic cost growth.The empirical observations are reported to confirm this theoretical prediction.
5 EXPERIMENTS
Experiments evaluate Actor-Mimic for multitask learning across Atari games and for transferring source-task knowledge to new targets. The method approaches expert performance on most source games and accelerates learning on several target games, but transfer can be weak or negative when tasks differ substantially or learning reaches a local maximum.
- Experimental setup: Actor-Mimic was evaluated on subsets of 20 Atari games, including 19 games where DQN had achieved super-human performance and Seaquest, where DQN performed poorly against humans.
- Multitask learning: The AMN reached close-to-expert performance on 7 of 8 games after about 5 million training frames, whereas expert networks used up to 50 million frames.
- Multitask learning: The AMN often had lower reward variance than expert DQNs, while achieving higher mean reward on Atlantis and relatively higher mean reward on Breakout and Enduro.
- Transfer learning: Transfer experiments initialized target-task DQNs with large AMNs trained using policy regression alone or combined policy and feature regression, comparing them with random initialization.
- Transfer learning: AMN pretraining increased learning speed on Breakout, Star Gunner, and Video Pinball, with positive transfer saving up to 5 million frames per game.Policy regression alone helped Breakout and Video Pinball, while feature regression additionally accelerated Star Gunner.
- Transfer limitations: Transfer was not pronounced for Krull and Road Runner, and multitask pretraining had no significant positive effect on Gopher or Robotank, with Robotank showing possible negative transfer.In Krull, policies tended toward an easily reached, difficult-to-escape high-reward local maximum.
6 RELATED WORK
Actor-Mimic connects model compression, imitation learning, and reinforcement-learning transfer methods. Its distinctive contribution is applying expert-guided policy and feature imitation to multitask deep reinforcement learning.
- Model compression: Model compression trains a smaller mimic network from one or more large expert models while attempting to preserve accuracy.
- Model compression: Actor-Mimic adapts model-compression guidance by matching expert policies and, in some variants, using expert intermediate features as training signals.
- Imitation learning: Actor-Mimic extends imitation learning from single-task expert imitation to a policy that learns from multiple task-specific experts.
- Imitation learning: Earlier expert-guided deep imitation approaches used MCTS policies, which required access to hidden emulator RAM and an environment model.
- Transfer learning: Related transfer methods use task-specific and task-independent state descriptors to learn source-task value functions before transferring them to a shared value function.
7 DISCUSSION
The paper presents Actor-Mimic as a single deep policy network trained across related source tasks, then reused to initialize learning on target tasks. Results support expert-level multitask behavior and faster target-task learning when source and target tasks are sufficiently similar.
- Actor-Mimic trains a single deep policy network over a set of related source tasks.
- The network reaches expert performance on many games simultaneously while retaining the model complexity of a single expert.
- Multitask pretraining can significantly improve learning speed on target tasks, indicating that source-task features can generalize to new tasks when sufficient similarity exists.
- The paper identifies targeted knowledge transfer as future work to select related source tasks and potentially reduce negative transfer.
APPENDIX A PROOF OF THEOREM 1
The proof establishes convergence of the iterative policy-and-parameter algorithm under irreducibility, aperiodicity, and Lipschitz assumptions. It bounds changes in stationary state distributions and softmax policies to obtain a contraction argument.
- Stationary-distribution bound: For any two policies, their stationary state distributions differ by at most a constant times the policy distance.
- Stationary-distribution bound: Transition-matrix differences are bounded by policy differences, which yields the stationary-distribution bound through an eigenvalue-dependent inequality.
- Softmax-policy bound: For softmax policies from the linear function approximator, policy-matrix distance is bounded by a constant times the distance between feature representations.
- Convergence theorem: Assuming an irreducible and aperiodic MDP for every induced policy and a Lipschitz-continuous Γ operator, policies and model parameters converge almost surely to a unique solution.
- Contraction argument: The proof uses these bounds to show that one algorithm iteration contracts policy distance by a factor β in [0, 1).
APPENDIX B AMN TRAINING DETAILS
Actor-Mimic Networks use Adam optimization and a shared 18-action output to support multitask Atari learning, while game-specific invalid actions are masked.
- AMNs are trained with the Adam optimization algorithm.
- Each AMN has one 18-unit output, with each unit corresponding to a possible Atari player action.
- During a game, invalid AMN actions are masked and the softmax is computed only over valid actions.
- DQN experiments use RMSProp, game-specific output layers, and the full 1,000,000-frame replay memory.
- The expert DQNs guiding AMNs follow the architecture, hyperparameters, and training procedure of Mnih et al. (2015).
APPENDIX C MULTITASK DQN BASELINE RESULTS
The multitask DQN baselines share either all network components or only convolutional layers, but Actor-Mimic trains more stably and performs consistently better across the eight games.
- MULTITASK DQN BASELINES: MDQN uses one shared DQN architecture and 18-action output across all eight games, masking invalid actions for each game.
- MULTITASK DQN BASELINES: MCDQN shares convolutional layers while assigning each game separate fully connected feature layers and action outputs.
- RESULTS: Actor-Mimic is more stable during training and consistently outperforms both MDQN and MCDQN across the evaluated games.
- RESULTS: MDQN and MCDQN often perform reasonably on a small subset of games while making little or no progress on others.
- RESULTS: MCDQN hardly improves results despite having substantially greater computational cost that scales linearly with the number of source games.
- ARCHITECTURES: The MDQN architecture matches the eight-game AMN architecture, while MCDQN adds eight parallel game-specific subnetworks above shared convolutions.
APPENDIX D ACTOR-MIMIC NETWORK MULTITASK RESULTS FOR TRANSFER PRETRAINING
Actor-Mimic transfer-pretraining experiments compare policy-only and feature-plus-policy objectives against smaller expert DQNs using 16 training epochs per game.
- TRANSFER NETWORK: The transfer network uses a larger architecture ending in 18 actions, with rectifier nonlinearities after every layer except the final one.
- AMN-POLICY: AMN-policy is trained solely with the policy regression objective for 16 epochs, or 4 million frames per game.
- COMPARISON: Both experiments compare Actor-Mimic training curves with smaller expert DQNs trained until convergence.
- EVALUATION: The evaluations report each expert DQN’s maximum test reward and mean testing reward over its final 10 epochs.
- AMN-FEATURE: AMN-feature is trained with both feature and policy regression objectives for the same 16-epoch, 4-million-frame-per-game schedule.
TABLE 1 BARPLOT
The barplot summarizes how the multitask Actor-Mimic compares with expert DQNs using relative mean and maximum reward improvements.
- The left plot shows relative mean reward improvement of the multitask AMN over expert DQNs.
- The right plot shows relative maximum reward improvement of the multitask AMN over expert DQNs.
- The learning-curve plots present the results reported in Table 2.