Source-linked AI summary
Dopamine: A Research Framework for Deep Reinforcement Learning
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, Marc G. Bellemare
TL;DR
Diverse deep RL research creates software needs that are difficult to anticipate and support with reusable code. The paper introduces Dopamine, a compact TensorFlow-based framework and taxonomy of research objectives, arguing that it addresses reproducibility and comparison challenges for fundamental deep RL research.
Problem
Growing diversity in deep RL research makes reusable software harder to design because agents are complex, algorithms are numerous, and future project needs are difficult to foresee.
Method
The paper introduces Dopamine, a compact TensorFlow-based framework with tested value-based agents, and reviews DQN research to develop a non-exhaustive taxonomy of deep RL objectives.
Results
Dopamine provides a stable, reliable, flexible, and reproducible framework for fundamental deep RL research, alongside a taxonomy highlighting heterogeneous research objectives.
Takeaways & Limitations
Different deep RL research objectives have different software needs, so frameworks designed for specific objectives can support reproducible and comparable research.
Takeaways & Limitations
The initial framework is limited to single-GPU, value-based agents running on the Arcade Learning Environment, with policy-based methods and other environments planned for future expansion.
Abstract
from arXiv · showhide
Deep reinforcement learning (deep RL) research has grown significantly in recent years. A number of software offerings now exist that provide stable, comprehensive implementations for benchmarking. At the same time, recent deep RL research has become more diverse in its goals. In this paper we introduce Dopamine, a new research framework for deep RL that aims to support some of that diversity. Dopamine is open-source, TensorFlow-based, and provides compact and reliable implementations of some state-of-the-art deep RL agents. We complement this offering with a taxonomy of the different research objectives in deep RL research. While by no means exhaustive, our analysis highlights the heterogeneity of research in the field, and the value of frameworks such as ours.
1 INTRODUCTION
Deep RL’s increasingly complex interactions and diverse research goals make reusable, comprehensive software harder to design. Dopamine addresses this challenge with a compact framework tailored to fundamental research and complemented by a taxonomy of research objectives.
- Motivation: Complex agent-environment interactions make reusable deep RL software more challenging to write.Agents increasingly comprise multiple interacting modules, while the growing algorithmic diversity makes future software needs difficult to anticipate.
- Contribution: Dopamine is a compact TensorFlow-based framework providing tested implementations of state-of-the-art value-based agents for the Arcade Learning Environment.Its initial version contains 12 Python files and includes interactive notebooks, trained models, and downloadable training data.
- Contribution: The framework prioritizes compactness over comprehensiveness to support fundamental deep RL research.The code is intended to remain understandable to newcomers while performing adequately for research at scale.
- Contribution: The paper argues that different deep RL research objectives require different software needs and positions Dopamine around algorithmic research and instruction.The taxonomy covers architecture research, comprehensive studies, visualization, algorithmic research, and instruction.
2 SOFTWARE FOR DEEP REINFORCEMENT LEARNING RESEARCH
The paper develops a non-exhaustive taxonomy of deep RL research objectives and relates each objective to distinct software requirements. It concludes that frameworks face a simplicity–complexity trade-off, with Dopamine deliberately favoring simplicity for algorithmic and instructional work.
- Research objectives: The taxonomy examines architecture research, comprehensive studies, visualization, algorithmic research, and instruction as distinct deep RL objectives.The objectives are not mutually exclusive, and the survey is explicitly not exhaustive.
- Software needs: Software needs are assessed by code reuse, code shelf life, performance value, and required complexity.The analysis asks whether existing code suffices, how long produced code remains useful, and how important high performance is.
- Architecture research: Architecture research typically requires complex frameworks with reusable modules and common interfaces spanning substantial code branches.Such frameworks may support engineering issues such as scaling distributed training, although early iterations can be discarded.
- Visualization: Visualization benefits from stable common tools but is often intrusive, difficult to reuse, and relatively insensitive to performance.The paper identifies a balance between simplicity and complexity as appropriate for visualization software.
- Algorithmic research: Algorithmic research benefits from simple frameworks because simple code facilitates radically different ideas and repeated iterations.Algorithmic changes range from modifying an equation to introducing new data structures.
- Instruction: Instructional software prioritizes simplicity and understandability, often adding notebooks, benchmark data, or visualizations for long-term use.Teaching-oriented code may sacrifice some performance to increase understandability.
- Conclusion: The taxonomy identifies a natural simplicity–complexity trade-off, and Dopamine chooses simplicity to facilitate algorithmic research and instruction.These design choices may empower some objectives while disadvantaging others.
3 DOPAMINE
Dopamine is a compact, self-contained, reliable, and reproducible TensorFlow framework focused initially on value-based reinforcement learning in the Arcade Learning Environment. It supports benchmarking and research through configurable settings, baseline agents, and shared experiment artifacts.
- Design principles: Dopamine is designed to be self-contained, compact, reliable, and reproducible for deep RL research.Its compactness and limited reliance on external libraries aim to lower the barrier to understanding the framework, while tests and shared artifacts support reliability and reproducibility.
- Scope: The initial framework focuses on value-based reinforcement learning applied to the Arcade Learning Environment.This restricted scope enabled design decisions centered on simplicity; computational performance is not the primary focus.
- Architecture: The complete codebase contains 12 files and a little over 2000 lines of Python code.The design includes components such as a Runner class for agent–ALE interaction and experiment bookkeeping.
- Experimental choices: Episode-termination choices materially affect reported performance, so Dopamine disables the Life Loss heuristic in its default settings.The framework documents both Game Over and Life Loss conditions; Life Loss can help some simpler games but hinder others.
- Configurations and baselines: Dopamine provides default and published gin-config settings for its agents, using uniform default hyperparameters as consistent baselines.The defaults combine agent hyperparameters from Hessel et al. with ALE parameters from Machado et al. and are intended to facilitate hyperparameter exploration rather than provide optimal settings.
- Reproducibility and benchmarking: Five independent runs across all 60 ALE games are distributed with checkpoints, TensorBoard event files, training logs, notebooks, and a performance webpage.These artifacts support benchmarking the provided agents against the default settings.
4 REVISITING THE ARCADE LEARNING ENVIRONMENT: A TEST CASE
The paper revisits how ALE episode termination, training measurement, and sticky actions affect reported deep RL performance. These parameter choices can change scores, while preserving some qualitative agent comparisons.
- Evaluation methodology: The study extends Machado et al.’s ALE methodology analysis from DQN to C51 and Rainbow, comparing reported performance under alternative environment parameters.The qualitative results remain the same when DQN is compared with either C51 or Rainbow.
- Episode termination: Life Loss adds artificial replay-memory episode boundaries after losing a life, whereas Game Over ends episodes when play would normally stop.Both termination definitions have been used in recent literature.
- Episode termination: Life Loss improves performance in some simpler games but hinders it in others because the agent cannot learn the true consequences of losing a life.Following Machado et al.’s guidelines, Dopamine disables Life Loss by default.
- Learning performance: Training and evaluation scores differ minimally for Dopamine’s default training and evaluation ϵ settings, supporting training-only learning curves.The comparison averages scores over 5 runs.
- Sticky actions: Sticky actions affect both Rainbow and DQN, typically reducing performance, while preserving their qualitative performance differences.In some cases, such as Rainbow on SPACE INVADERS, sticky actions improve performance; Dopamine enables them by default.
5 RELATED WORK
Dopamine is positioned within a growing ecosystem of deep RL frameworks and open-sourced agents. Existing offerings emphasize comprehensive algorithms, specialized environments, distributed execution, or individual-agent reproducibility, while the paper concludes Dopamine occupies a unique niche.
- Dopamine’s niche: The review concludes that Dopamine fills a unique niche in the deep reinforcement learning software ecosystem.The paper frames this conclusion as its perspective on the interplay between frameworks and the research-objective taxonomy.
- Deep RL frameworks: OpenAI Baselines provides comprehensive deep RL implementations focused particularly on single-threaded, policy-based algorithms, alongside related libraries such as Coach, Tensorforce, and Keras RL.Other frameworks target continuous control, distributed reinforcement learning, or real-time strategy games.
- Deep RL frameworks: Frameworks offering many agents and algorithms are well positioned for architecture research and comprehensive studies.The reviewed ecosystem includes RLLab, RLLib, and ELF, which emphasize different environments or execution settings.
- Open-sourced agents: Individual agents such as DQN, PPO, and IMPALA have also been open-sourced to support reproducibility and disseminate implementation knowledge.These offerings are distinct from broader multi-agent frameworks.
6 CONCLUSION AND FUTURE WORK
The paper concludes that Dopamine offers a stable, reliable, flexible, and reproducible framework for fundamental deep RL research. Its initial compact design deliberately limits scope to single-GPU, value-based agents on the ALE, with broader extensions left for future work.
- Conclusion: Dopamine provides a stable, reliable, flexible, and reproducible framework for fundamental deep reinforcement learning research.The authors connect this design to concerns about making new research reproducible and easy to compare against.
- Future work: The initial offering focuses on single-GPU, value-based agents running on the Arcade Learning Environment to keep the framework simple and compact.Future plans include policy-based methods, other environments, cautious consideration of distributed methods, and advanced visualization tools.
A PERFORMANCE STATISTICS FOR DOPAMINE
The paper notes that computational performance is not Dopamine’s focus, while still providing performance statistics.
- Performance statistics: Dopamine provides performance statistics despite not focusing on computational performance.The statistics appear in the appendix.
A.1 RUNTIME
Runtime performance varies across agents and games, so Dopamine reports two measured extremes on a Tesla P100 GPU.
- 800 fps is the reported runtime for DQN on Pong.
- 371 fps is the reported runtime for IQN on Asterix.
A.2 DISKSPACE
Dopamine checkpoints for recovery while retaining only recent checkpoints, with replay-buffer disk usage varying substantially across agent–game pairs.
- The replay buffer is configured by default to hold 1 million frames and is stored as compressed NumPy objects when checkpointing.
- 4.3Kb is the compressed replay-buffer size reported for DQN on Pong.Pong has few active pixels per frame, which the paper gives as the reason this footprint is unsurprising.
- 1.2Gb is the compressed replay-buffer size reported for IQN on YarsRevenge.The paper attributes this larger footprint to a pseudo-random strip running down the screen.
- The interactive notebook visualizes an agent’s performance against trained baselines after training.
C CREATING A NEW AGENT FROM SCRATCH
Dopamine demonstrates creating agents from scratch with the minimum Runner-facing functionality, using illustrative examples and configurable experiment settings.
- Creating an agent: A from-scratch agent must implement episode initialization, episode termination, stepping, action selection, and checkpoint-related methods.
- Creating an agent: The StickyAgent randomly selects an action and retains it, switching actions with probability 0.1.
- Running examples: The example Runner uses small step counts and terminates after 110 iterations instead of the standard 200 to demonstrate partial-run plotting.
- Configuration: Dopamine provides gin configuration files for agent settings, including GPU device assignments and settings corresponding to prior studies.
- Configuration: Rainbow configurations modify prior hyperparameters as necessary to match Rainbow settings for apples-to-apples comparison.
D.4.1 DEFAULT SETTINGS
The default settings include IQN-specific configuration choices, including a one-million-step epsilon decay period and sticky actions with probability 0.25.
- The IQN configuration follows Dabney et al. (2018) while matching Rainbow settings where necessary for apples-to-apples comparison.
- The default IQN configuration uses prioritized replay imports and sets sticky actions with probability 0.25.
- IQN uses a one-million-agent-step epsilon decay period in the listed settings.