Source-linked AI summary
Deep Attention Recurrent Q-Network
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, Anastasiia Ignateva
TL;DR
DQN achieved strong Atari control, but its training cost and limited mechanisms motivated extensions. This paper integrates LSTM and soft or hard visual attention into DQN, finding that DARQN surpasses original DQN on some Atari games while exposing the regions guiding decisions.
Problem
DQN’s long training time and need for improved learning abilities motivate extensions using recurrent memory and visual attention.
Method
DARQN integrates LSTM and soft or hard visual attention into DQN, using focused image regions to produce recurrent context for action-value evaluation.
Results
DARQN surpasses original DQN on at least some of five Atari 2600 games despite having fewer optimized parameters, while visualizations show the regions receiving attention.
Takeaways & Limitations
Attention-based DARQN provides a compact Atari control model whose visual focus offers insight into the agent’s decision process.
Takeaways & Limitations
The simple policy-gradient training used for hard attention performs relatively poorly, motivating alternative methods for training stochastic attention networks.
Abstract
from arXiv · showhide
A deep learning approach to reinforcement learning led to a general learner able to train on visual input to play a variety of arcade games at the human and superhuman levels. Its creators at the Google DeepMind's team called the approach: Deep Q-Network (DQN). We present an extension of DQN by "soft" and "hard" attention mechanisms. Tests of the proposed Deep Attention Recurrent Q-Network (DARQN) algorithm on multiple Atari 2600 games show level of performance superior to that of DQN. Moreover, built-in attention mechanisms allow a direct online monitoring of the training process by highlighting the regions of the game screen the agent is focusing on when making decisions.
1 Introduction and Related Work
The paper extends DQN to address limited temporal memory and long training times by combining recurrent memory with visual attention. The authors motivate attention as a way to focus computation on informative image regions while exposing where the agent is looking.
- Proposed direction: The proposed approach adds LSTM and visual attention mechanisms to DQN, the deep network used as an action-utility function in DQL.
- Existing limitations: DQN bases actions on the last four game states, limiting games that require remembering events farther in the past.
- Existing limitations: DQN training takes 12-14 days on a GPU, making experiments with alternative architectures and parameter settings costly.
- Proposed direction: Attention selects and focuses on small informative image regions, reducing parameters and training and testing computations.
- Proposed direction: Attention-based models also let researchers visualize where and what the agent focuses on during Deep Q-Learning.
2 Deep Attention Recurrent Q-Network
DARQN combines convolutional, attention, and recurrent networks to transform visual game states into action values and the next attention context. It supports soft attention through weighted feature aggregation and hard attention through stochastic location sampling, with end-to-end Q-learning and policy-gradient training.
- Architecture: CNN extracts D feature maps from the current visual state, while attention converts these maps into vectors and a context vector z_t.
- Architecture: The LSTM receives the context vector and previous recurrent states, then produces action Q-values and the next attention context.
- Soft attention: Soft attention computes z_t as a weighted sum of all location vectors, with weights produced by a two-layer network followed by softmax.
- Training: DARQN minimizes a sequence of Q-learning losses using approximate target values, rewards, discounting, and an ε-greedy behavior distribution.
- Training: The model is differentiable and trained end-to-end, while hard-attention training combines gradients from the attention and recurrent pathways in convolutional layers.
- Hard attention: Hard attention samples one of L locations using a stochastic policy and trains its parameters with REINFORCE based on future discounted returns.
3 Experiments
The experiments compare DARQN with DQN and DRQN across five Atari games, examining architecture, training behavior, and learned attention regions. DARQN performs well on some games but does not consistently surpass DQN, with clear weaknesses on Breakout and hard attention in Seaquest.
- Evaluation setup: DARQN was evaluated against DQN and DRQN on Breakout, Seaquest, Space Invaders, Tutankham, and Gopher.The comparison used the authors’ DARQN implementation alongside Torch DQN and Caffe DRQN implementations.
- Network architecture: 845,428 and 845,171 adjustable parameters were used by hard and soft DARQN on Seaquest, versus 1,693,362 for both DQN and one-step DRQN.Seaquest has 18 possible actions in this comparison.
- Attention analysis: Soft attention visualizations show focus on Breakout’s ball trajectory and a shift in Seaquest from the oxygen indicator and nearby enemies to the submarine during resurfacing.The attention regions were visualized by upsampling subsidiary feature maps through the CNN layers.
- Evaluation setup: The evaluation measured average reward per episode during 5M training steps, with assessments every 50,000 steps.The evaluation policy was ε-greedy with ε = 0.05 and ran for 25,000 steps.
- Performance results: DARQN did not outperform DQN and DRQN on every game; Figure 2 therefore compares training on the games where DARQN performed best and worst.The main five-game comparison is reported in Table 1.
- Performance results: On Seaquest, both DARQN variants performed strongly, but hard attention failed to learn the need for regular resurfacing, possibly because policy-gradient training can converge to a local optimum.The soft attention model outperformed the hard attention model in this respect.
- Performance results: On Breakout, increasing DARQN unroll steps from 4 to 10 improved performance somewhat, but neither soft nor hard DARQN surpassed DQN.Figure 3 reports reward across training epochs; one epoch corresponds to 50,000 steps.
- Attention analysis: Hard attention visualizations show immediate response to temporary ball disappearance in Breakout and sustained focus on a Seaquest enemy until its destruction.These examples provide qualitative views of the regions selected by the hard attention model.
4 Conclusion and Future Work
The paper integrates attention mechanisms into DQN and reports that the resulting model surpasses original DQN on some Atari games despite having fewer optimized parameters. It also identifies limitations of hard-attention training and proposes directions involving multi-scale or glimpse attention and improved stochastic optimization.
- Conclusion: DARQN integrates attention mechanisms into Deep Q-Network and, on five Atari 2600 games, surpasses original DQN on some games despite fewer optimized parameters.The authors interpret this as evidence of greater generalization ability.
- Conclusion: Attention-based DARQN displays the game-screen regions receiving focus during decisions, providing insight into the agent’s behavior.
- Future Work: The attention mechanism acts as an additional LSTM filter gate for CNN-produced structured visual data from the entire image.
- Future Work: Future work includes testing multi-scale or glimpse attention mechanisms within DQN.
- Future Work: The policy-gradient algorithm used to train hard-attention DARQN performed relatively poorly, motivating methods to reduce stochastic-gradient variability and improve stochastic attention training.