PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Recurrent Q-Learning for Partially Observable MDPs

Matthew Hausknecht, Peter Stone

arXiv:1507.06527v4cs.LG

TL;DR

The paper addresses the limited memory and incomplete observations of DQN agents in partially observable environments. It replaces DQN’s first post-convolutional fully connected layer with an LSTM, creating DRQN, and evaluates its robustness across observation conditions. DRQN matches DQN on standard settings while adapting better when observation quality changes, although recurrency provides no systematic learning advantage over frame stacking.

  • Problem

    DQN relies on a limited history of screens, while important state information can be missing from individual observations in partially observable tasks.

  • Method

    DRQN minimally modifies DQN by replacing its first post-convolutional fully connected layer with a recurrent LSTM that integrates observations over time.

  • Results

    DRQN generalizes better than DQN across changing observation quality, scaling with observability when trained on partial observations and degrading less under partial evaluation.

  • Takeaways & Limitations

    Given the same history length, recurrency is a viable alternative to stacking frames and is especially useful when observation quality changes at evaluation.

  • Takeaways & Limitations

    Across ten Flickering MDPs and non-flickering Atari games, recurrency showed no systematic performance benefit over stacking observations.

Abstract

from arXiv · show

Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting \textit{Deep Recurrent Q-Network} (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

Introduction

DQN agents rely on limited histories of game screens, creating partial observability when important state information lies beyond that window. The paper introduces recurrency to estimate hidden state from observation sequences and improve handling of incomplete information.

  • Problem: DQN uses a limited history of past game screens, so tasks requiring memory beyond four frames become partially observable.The future can depend on events older than the network’s current input.
  • Problem: A single screen can omit crucial dynamics such as object velocity, making optimal action selection difficult.In Pong, the screen shows the ball’s location but not its direction of travel.
  • Approach: DRQN adds recurrency to Deep Q-Learning so sequences of observations can better estimate the underlying system state.The recurrent network narrows the gap between observation-based and state-based Q-value estimates.
  • Background: Deep Q-Learning estimates long-term action returns with neural networks that approximate Q-values over large state-action spaces.DQN updates network parameters using replayed experiences and a separate target network for more stable learning.
  • Background: POMDPs model settings where agents receive only partial glimpses of the underlying system state rather than complete Markov states.This formalism better captures many real-world environments with incomplete state information.

DRQN Architecture

DRQN minimally modifies DQN by replacing its first post-convolutional fully connected layer with an LSTM. A single preprocessed frame is transformed through convolutions, recurrent processing, and a final action-value layer.

  • Architecture: DRQN replaces DQN’s first fully connected layer with a same-sized recurrent LSTM to isolate the effects of recurrency.The convolutional and recurrent parameters are learned jointly from scratch.
  • Architecture: A single 84 × 84 preprocessed image passes through three convolutional layers before entering the LSTM.The LSTM processes convolutional activations across timesteps.
  • Architecture: The LSTM output is passed through a linear layer that produces one Q-value for each available action.Figure 2 depicts the final two timesteps of this temporal processing pipeline.

Stable Recurrent Updates

DRQN training compares sequential and randomized recurrent updates, trading hidden-state continuity against compatibility with DQN’s random replay sampling. Both strategies produce convergent policies with similar performance, so the experiments use randomized updates.

  • Update strategies: Random updates sample episodes and starting points randomly, unroll for a fixed number of timesteps, and zero the LSTM state at each update.Targets are generated from the target Q-network.
  • Update strategies: Sequential updates carry the LSTM hidden state through an episode but violate DQN’s random experience-sampling policy.They begin at the episode start and proceed through the episode.
  • Update strategies: Zeroing the hidden state makes it harder for the LSTM to learn dependencies spanning longer than the backpropagation-through-time horizon.This is the principal temporal limitation of randomized updates.
  • Experimental choice: Both update strategies yield convergent policies with similar performance across the evaluated games, motivating randomized updates for all reported results.The authors expect the results to generalize to sequential updates.

Atari Games: MDP or POMDP?

Atari game screens provide incomplete observations of the underlying console state, but stacking the last four screens allows DQN to infer a fuller state representation for the games studied.

  • State observability: A single Atari game screen can be insufficient to determine the underlying system state observed through console RAM.Humans and agents see game screens rather than the full 128-byte console RAM state.
  • State observability: DQN expands its input to the last four game screens, making the investigated games MDPs rather than POMDPs under that representation.The authors found no partially observable games among the 49 examined with four-frame input.

Flickering Atari Games

Flickering Atari tasks test whether agents can integrate information across incomplete observations. DRQN succeeds with one frame per timestep by using recurrence to recover temporal and event information unavailable to its convolutional layers alone.

  • Flickering task: Flickering Pong obscures each screen with probability p = 0.5, creating a POMDP that requires integrating observations across frames.Success requires estimating ball location, velocity, and paddle location despite potentially contiguous obscured inputs.
  • Frame history: A 10-frame DQN can convolutionally detect object velocity from screen history, although less reliably than in unobscured Pong.The learned filters respond to movement patterns in the game objects.
  • Learned representations: Figure 3 shows early convolutional filters detecting paddles, deeper filters tracking ball-paddle interactions, and LSTM units detecting high-level Pong events.The visualizations use the last 10 frames, with more luminance assigned to recent frames.
  • Recurrent integration: DRQN performs well with only one input frame per timestep, requiring its recurrent layer to integrate noisy information through time because convolutional layers cannot detect velocity from a single frame.LSTM units detect events including misses and ball reflections.
  • Comparison: With ten-step backpropagation through time, recurrent 1-frame DRQN and non-recurrent 10-frame DQN receive the same history length, making recurrence a viable alternative to frame stacking.The comparison isolates how temporal information is integrated rather than changing the available history length.

Evaluation on Standard Atari Games

On standard Atari games, DRQN performs roughly on par with DQN rather than systematically outperforming it. Its results vary by game, excelling on Frostbite and Double Dunk but struggling on Beam Rider.

  • Evaluation set: The evaluation covers nine Atari games spanning flickering sprites, shooters, platforming, sports, timing, and object-rich gameplay.Asteroids and Double Dunk were identified as potential recurrent-learning candidates because of naturally flickering sprites.
  • Standard Atari results: With four-frame inputs, the games are MDPs rather than POMDPs, so the experiments provide no general reason to expect DRQN to outperform DQN.This setup tests recurrence where the stacked input already supplies the relevant state history.
  • Standard Atari results: On average, DRQN performs roughly as well as DQN, outperforming it on Frostbite and Double Dunk but performing significantly worse on Beam Rider.The table caption summarizes the same game-level pattern.
  • Game-level variation: Frostbite performance jumps after DRQN learns a policy that reliably advances beyond the first level, whereas Beam Rider represents its worst performance.The Frostbite improvement occurs after 12,000 episodes.

MDP to POMDP Generalization

When trained on standard games and evaluated on flickering equivalents, DRQN retains more performance than DQN across all levels of missing information.

  • MDP to POMDP Generalization: DRQN captures more of its previous performance than DQN across all levels of flickering when trained with full observations and evaluated with partial observations.Both algorithms incur significant performance decreases from missing information.
  • MDP to POMDP Generalization: Figure 5 reports average percentages of the original game score across all 9 games in Table 1.

Related Work

The paper situates DRQN among recurrent approaches to partial observability, distinguishing its temporal-difference learning and pixel-based Atari setting from prior work.

  • Related Work: Unlike prior policy-gradient approaches, DRQN uses temporal-difference updates to bootstrap an action-value function.Its convolutional and LSTM layers are jointly trained directly from pixels without hand-engineered features.
  • Related Work: Earlier LSTM work solved partially observable corridor and cartpole tasks better than comparable non-LSTM RNNs in tiny state spaces.
  • Related Work: Related work combined LSTM with deep reinforcement learning for text-based fantasy games whose underlying state spaces were relatively low-dimensional.The more complex cited game featured only 56 underlying states.

Discussion and Conclusion

DRQN combines LSTM recurrence with DQN to integrate partial observations and generalize across changing observability. Across the examined games, however, recurrency provides no systematic learning advantage over frame stacking.

  • Discussion and Conclusion: DRQN combines an LSTM with DQN to handle noisy observations in POMDPs while integrating information across frames from a single frame per timestep.
  • Discussion and Conclusion: When trained with partial observations, DRQN generalizes to complete observations, with Flickering Pong performance scaling with observability.Performance reaches near-perfect levels when every game screen is observed.
  • Discussion and Conclusion: When trained on standard Atari games and evaluated on flickering games, DRQN generalizes better than DQN at all levels of partial information.
  • Discussion and Conclusion: Across ten Flickering MDPs and non-flickering Atari games, experiments show no systematic improvement from recurrency compared with stacking observations.Pong appears to be an outlier among the examined games.

Appendix A: Alternative Architectures

Alternative architecture experiments favored replacing DQN’s first fully connected layer with an LSTM, while larger recurrent and stacked-frame models raised computational costs.

  • Appendix A: Alternative Architectures: Replacing IP1 with an LSTM was favored because it gives the LSTM direct access to convolutional features.Adding a Rectifier layer after the LSTM consistently reduced performance.
  • Appendix A: Alternative Architectures: 709% percent improvement was obtained when the LSTM replaced IP1, compared with 533% for ReLU-LSTM replacing IP1 and 418% for LSTM over IP1.ReLU-LSTM over IP1 achieved 0%.
  • Appendix A: Alternative Architectures: Combining four-frame stacking with an LSTM unrolled over 10 timesteps added parameters and training time without improving results.
  • Appendix A: Alternative Architectures: A model unrolled for 30 iterations with 10 stacked frames would require over 56 days to reach 10 million iterations.

Appendix C: Experimental Details

The experiments used the Arcade Learning Environment with specified options and implementation settings, including recurrent-training stabilization. Computational cost was organized by input-frame count and LSTM unroll length, alongside flickering-game evaluations.

  • Training ran for 10 million iterations with a replay memory of 400,000, while policies were evaluated every 50,000 iterations over 10 episodes.
  • LSTM gradients were clipped to ten, and all networks used ADADELTA with learning rate 0.1 and momentum 0.95.
  • Networks were trained in the Arcade Learning Environment using color averaging, the minimal action set, and death detection.
  • Table 2 reports average milliseconds per backward/forward pass for different input-frame counts and LSTM backpropagation unrolls of 1, 10, or 30 steps.
  • DRQN was implemented in Caffe, and its source was reported as available at a redacted location.
  • Table 3 evaluates partially observable flickering game equivalents in which each screen is obscured with probability 0.5, with bold values marking statistical significance.
Loading 1507.06527v4…