Source-linked AI summary
On Improving Deep Reinforcement Learning for POMDPs
Pengfei Zhu, Xin Li, Pascal Poupart, Guanghui Miao
TL;DR
Deep reinforcement learning has seen little use in partially observable environments, where DQNs’ full-observability assumption is limiting. ADRQN recurrently processes action-observation pairs, and the reported comparisons show improved performance over DRQN and DDRQN in partially observable settings.
Problem
DQNs assume full state observability, limiting their suitability for POMDPs where arbitrarily long observation histories may be needed for action selection.
Source: §1 · p. 1, §2.1 · p. 2Method
ADRQN encodes actions and pairs them with observations, then uses an LSTM to infer latent states for Q-value computation.
Source: §1 · p. 2, §2.3 · p. 3Results
ADRQN matches DRQN and DDRQN in full-observation settings and shows clear improvements over them in partially observable settings across the reported Atari comparisons.
Source: §4.2 · p. 5Takeaways & Limitations
The paper reports ADRQN’s effectiveness on several POMDP problems compared with state-of-the-art approaches.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architecture called Action-specific Deep Recurrent Q-Network (ADRQN) to enhance learning performance in partially observable domains. Actions are encoded by a fully connected layer and coupled with a convolutional observation to form an action-observation pair. The time series of action-observation pairs are then integrated by an LSTM layer that learns latent states based on which a fully connected layer computes Q-values as in conventional Deep Q-Networks (DQNs). We demonstrate the effectiveness of our new architecture in several partially observable domains, including flickering Atari games.
1 Introduction
Deep Q-learning has succeeded in fully observable tasks, but DQNs’ full-observability assumption makes them unsuitable when optimal action selection depends on long observation histories. ADRQN addresses this gap by processing action-observation pairs recurrently.
1 Introduction
- Deep reinforcement learning has tackled high-dimensional tasks, and DQNs play Atari games using the past four frames as input.The introduction contrasts these settings with partially observable tasks that may require arbitrarily long observation histories.Source: §1 · p. 1
- In POMDPs, DQNs’ full-observability assumption is inadequate when arbitrarily long observation histories are needed for optimal action selection.Source: §1 · p. 1
- Prior deep-RL approaches have distinct constraints: DBQN requires known environment dynamics, while DRQN omits action histories and DDRQN decouples actions from observations.The introduction says this decoupling negatively affects belief inference.Source: §1 · pp. 1–2
- ADRQN encodes actions, pairs them with observations, and feeds the sequence to an LSTM that learns latent states for Q-value computation.Source: §1 · p. 2
- The introduction reports ADRQN’s effectiveness in several Atari 2600 games and points to Table 1 for comparisons with other deep Q-learning approaches.Source: §1 · p. 2, Table 1 · p. 2
2 Background
The background reviews value-based deep RL and explains why partial observability challenges methods that rely on short observation histories. It introduces POMDPs and recurrent Q-networks as context for incorporating history into learning.
2 Background
- The section situates DQNs and POMDPs within the broader framework of sequential decision-making and reviews the recurrent approach to partial observability.The background covers DQNs, POMDPs, and DRQNs.Source: §2 · p. 2
2.1 Deep Q-Networks
- MDPs assume fully observable states, while DQNs approximate Q-values with a neural network and can be inaccurate when states are partially observable.DQN relies on full state observability, an assumption that limits its use when observations omit relevant state information.Source: §2.1 · p. 2
2.2 Partially Observable Markov Decision
- POMDPs represent partial observability through beliefs over states, updated using the previous belief, action, and next observation.The value function over beliefs can be represented by linear α-vectors whose maximum envelope forms the value function.Source: §2.2 · pp. 2–3
2.3 DRQN
- DRQN replaces DQN’s post-convolutional fully connected layer with an LSTM that integrates arbitrarily long observation histories to estimate state.DRQN matches DQN on standard MDPs and outperforms it in partially observable domains, but does not explicitly incorporate actions into its histories.Source: §2.3 · p. 3
3 ADRQN - Action-specific Deep Recurrent Q-Network
ADRQN processes action-observation pairs through an LSTM and uses its latent states to compute Q-values. Its design embeds actions to balance their representation with observation features, stores sequential transitions, and limits training unrolls to ten steps.
3 ADRQN - Action-specific Deep Recurrent Q-Network
- ADRQN couples each performed action with its observation, processes the resulting sequence with an LSTM, and computes Q-values from its latent states.The illustrated Atari network maps the action to a 512-D vector and produces 18 Q-values after processing the frame and LSTM output.Source: §3 · p. 3, Figure 1 · p. 4
- A fully connected layer embeds one-hot actions into 512 dimensions to reduce the imbalance with much higher-dimensional observation features.Direct concatenation performed poorly because the action and observation vectors differed greatly in length, which led to numerical instability.Source: §3 · p. 3
- Replay stores sequential transitions containing the previous action, current observation, chosen action, reward, and next observation.These histories let the LSTM receive sequences of action-observation pairs during decisions and network updates.Source: §3 · p. 3
- Training unrolls the LSTM for 10 time steps, balancing longer history coverage against experiment efficiency and matching DRQN’s setting.The authors note that integrating an entire Atari episode would require thousands of transitions and substantially increase training cost.Source: §3 · pp. 3–4
- For updates, the algorithm samples transition sequences, computes Q-values for the Q-network and target network, and backpropagates their difference.Episodes initialize the first action to no-op and the first hidden-layer input to zero; the target network is initialized separately.Source: §3 · p. 4
4 Experiments
Experiments assess ADRQN on Atari games and flickering variants during training, testing, and changes in observation probability. Across these settings, it improves over DRQN and DDRQN most clearly under partial observability.
4.1 Experiments setup
- Training uses replay-sampled 10-step action-observation sequences, zero-initialized LSTM state, frame skip 4, and reward clipping to ±1.Source: §4.1 · pp. 4–5, §4.2 · p. 5
4.2 Training
- During training, ADRQN matches or slightly exceeds the baselines with full observations and improves clearly with partial observations, especially in Frostbite.Source: Figure 2 · p. 5, Figure 3 · p. 5, §4.2 · p. 5
4.3 Testing Evaluation
- With half the frames obscured, ADRQN significantly outperforms DRQN and DDRQN in all five games, matching full-observation scores in Chopper Command and Frostbite.Source: Table 3 · p. 6, §4.3 · p. 6, Table 2 · p. 6
4.4 Generalization Performance
- After training at observation probability 0.5, ADRQN outperforms DRQN and DDRQN across tested observation probabilities in Pong and Frostbite; after full-observation training, it is relatively robust as obscuration rises.Source: Figure 4 · p. 6, §4.4 · p. 6, Figure 5 · p. 6
5 Brief Discussion
The discussion argues that recurrent models truncate observation histories for training efficiency, so prior actions cannot be reliably recovered from the remaining observations. It motivates explicitly including action-observation pairs during training.
5 Brief Discussion
- Recurrent-model training truncates histories, preventing recovery of the proper action sequence from the remaining observations.The discussion contrasts this with a fixed deterministic policy, where actions could be inferred from the entire observation sequence.Source: §5 · p. 6
- For complex partially observable tasks, the observation set is infinite, making action inference from finite histories harder.Source: §5 · p. 6
- The discussion suggests that explicitly incorporating coupled action-observation pairs would help, especially during training.Source: §5 · p. 6
6 Conclusion
The paper reports ADRQN's effectiveness on several POMDP problems compared with state-of-the-art approaches. Its architecture integrates action-observation pairs with an LSTM to infer latent states for Q-value computation.
6 Conclusion
- ADRQN encodes actions and couples them with CNN-extracted observation features to form action-observation pairs.Source: §6 · p. 6
- An LSTM integrates the pairs to infer latent states, and a fully connected layer computes Q-values to guide learning.Source: §6 · p. 6
- ADRQN is demonstrated on several POMDP problems in comparison to state-of-the-art approaches.Source: §6 · p. 7