Source-linked AI summary
Graying the black box: Understanding DQNs
Tom Zahavy, Nir Ben Zrihem, Shie Mannor
TL;DR
Deep reinforcement learning agents are difficult to understand and configure, especially when learning from high-dimensional inputs. The paper introduces feature-based analysis tools and SAMDP to identify spatio-temporal abstractions in DQNs. It finds hierarchical state aggregation and options, enabling policy interpretation and debugging while revealing non-Markovian and task-specific limitations.
Problem
Deep reinforcement learning lacks tools for understanding learned agents and requires difficult choices over architectures, hyperparameters, and environment modeling.
Method
The paper combines feature-based clustering with an automatically learned Semi Aggregated Markov Decision Process to identify spatio-temporal abstractions.
Results
The analysis reveals hierarchical state aggregation and options in DQN features and describes learned policies across three Atari2600 games.
Takeaways & Limitations
Policy interpretation and debugging can clarify learned behavior, while SAMDP can improve an existing policy without retraining.
Takeaways & Limitations
SAMDPs are not necessarily Markovian, and the analysis identifies a Breakout local-optimum failure on states with few remaining bricks.
Abstract
from arXiv · showhide
In recent years there is a growing interest in using deep representations for reinforcement learning. In this paper, we present a methodology and tools to analyze Deep Q-networks (DQNs) in a non-blind matter. Moreover, we propose a new model, the Semi Aggregated Markov Decision Process (SAMDP), and an algorithm that learns it automatically. The SAMDP model allows us to identify spatio-temporal abstractions directly from features and may be used as a sub-goal detector in future work. Using our tools we reveal that the features learned by DQNs aggregate the state space in a hierarchical fashion, explaining its success. Moreover, we are able to understand and describe the policies learned by DQNs for three different Atari2600 games and suggest ways to interpret, debug and optimize deep neural networks in reinforcement learning.
1. Introduction
Deep reinforcement learning addresses difficult high-dimensional control problems, but its learned representations and configuration choices remain hard to understand. This paper introduces analysis tools showing how DQNs form abstractions and support interpretation, debugging, and optimization.
- Motivation: High-dimensional inputs create a longstanding reinforcement-learning challenge because conventional solutions require problem-specific state representations.These engineered representations reduce flexibility and make learning more tedious.
- Motivation: DQN improved Atari learning stability and demonstrated a robust ability to learn representations.Its target network and Experience Replay contributed to training stability.
- Motivation: Deep reinforcement learning requires difficult outer-loop choices over architectures, hyperparameters, optimization algorithms, and MDP modeling.Practitioners commonly rely on trial and error or exhaustive configuration searches.
- Approach: The paper discovers DQN’s internal model through hand-crafted clustering and an automatically learned Semi Aggregated Markov Decision Process.These approaches identify spatio-temporal abstractions directly from learned representations.
- Findings: The analysis finds hierarchical state aggregation and options in DQNs, indicating that abstractions are learned automatically rather than manually specified.The authors connect this finding to an explanation of deep reinforcement learning’s success.
- Applications: The proposed tools interpret learned policies and provide debugging methods that reduce deep-reinforcement-learning hyperparameter grid search.Examples address game modeling, termination and initial states, and score over-fitting.
2. Background and Related Work
The background reviews DQN mechanics, visualization methods, and spatial and temporal abstractions for simplifying reinforcement-learning analysis. It motivates SAMDP as a combination of state aggregation and skills while noting important limitations of existing abstraction models.
- Deep Q Networks: DQN approximates the optimal action-value function with a convolutional neural network and trains by minimizing Bellman temporal-difference error.Experience Replay stores experience tuples for later training, while a target network is updated periodically.
- Deep Q Networks: DQN clips rewards to [−1, 1], uses separate online and target networks, and represents states with sequences of history frames.These design choices support training stability and capture game dynamics.
- Visualization: t-SNE reduces high-dimensional representations to a visual map that can reveal structure across multiple scales and sub-manifolds.The paper uses it to visualize and interpret structures learned by the network.
- Abstractions: Temporal abstractions extend MDPs with skills or options defined by initiation states, intra-skill policies, and termination probabilities.SMDP analysis shortens the planning horizon but requires identifying a potentially difficult skill set.
- Abstractions: State aggregation replaces MDP states with clusters sharing attributes such as similar policies or value functions.It reduces model dimensions, but does not reduce the action space and may produce non-Markovian aggregated models.
3. Methods
The methodology combines visualization, feature-based clustering, and SAMDP modeling to analyze DQN representations and policies through spatio-temporal abstractions.
- Manual clustering: The visualization tool maps states with t-SNE, colors them using global or hand-crafted features, and exposes corresponding game images and saliency maps.Users can navigate between states along trajectories.
- Manual clustering: Collected statistics include activations, Q values, termination signals, rewards, actions, and emulator-derived features used to color and filter t-SNE states.Filtering reveals patterns that are otherwise difficult to see.
- Manual clustering: The analysis visualizes learned policy dynamics as transitions in a three-dimensional t-SNE representation and uses saliency maps to identify influential input pixels.Transitions are displayed with arrows, while saliency maps are computed from the network Jacobian.
- Manual clustering: Manual clustering identifies common cluster attributes, hierarchical state aggregation, and options with recognizable entrance and termination areas.Some options support inferred initiation and termination rules.
- Semi Aggregated MDP: SAMDP combines spatial aggregation and temporal skills, reducing state-space dimensions and planning horizon through five stages from feature selection to model selection.The stages are feature selection, aggregation, skill identification, inference, and model selection.
- Semi Aggregated MDP: The clustering procedure uses candidate ensembles and temporally coherent K-means assignments, requiring neighboring trajectory states to remain close to the same cluster mean.Candidate models are evaluated with spatio-temporal criteria, while inertia, transition intensity, and entropy support model selection.
- Semi Aggregated MDP: SAMDP skills are defined by initiation and termination clusters, with skill duration inferred from observed trajectories rather than fixed in advance.The model also infers skill rewards and transition probabilities from observations.
4. Experiments
The experiments apply manual and SAMDP analyses to a gridworld and three Atari games, showing how transformed representations expose skills and policy structure.
- 4. Experiments: Experiments cover Gridworld, Pacman, Seaquest, and Breakout, analyzing each domain with hand-crafted features, manual clustering, and SAMDP modeling.The games span strong, weak, and popular DQN performance settings.
- 4.1 Gridworld: Figure 3 contrasts MDP, SMDP, AMDP, and SAMDP diagrams, with SAMDP intra-cluster transitions explaining skills and inter-cluster transitions explaining policy.Primitive actions connect individual MDP states, whereas skills connect aggregated SAMDP states.
- 4.1 Gridworld: In Gridworld, the MDP representation contains many states and obscures the skills used by the agent.The domain requires reaching a ball and returning to the origin.
- 4.1 Gridworld: The transformation φ flips and translates states after the ball is reached, disentangling routes to and from the ball.Here L denotes the maze width.
- 4.1 Gridworld: SAMDP clusters align with skills after transformation, making behavior interpretable through intra-cluster and inter-cluster transitions.The gridworld result indicates that the MDP hierarchy is discovered under a suitable representation.
4.2 Setup for Atari
Across Atari experiments, the analysis combines neural-activation visualization, handcrafted features, clustering, and SAMDP modeling to expose hierarchical policies and temporally extended skills. In Breakout, these tools identify tunnel-carving and post-tunnel behavior, while also revealing a local-optimum failure.
- Setup: Handcrafted game features describe entities and dynamics, including paddle and ball positions, movement direction, missing bricks, and tunnel structure.A tunnel is approximated by detecting at least one clear brick column.
- Breakout: Breakout clusters reveal a hierarchical policy: the agent carves a left tunnel, then keeps the ball above the bricks as long as possible.Clusters 1–3 form a landmark option for tunnel carving, while clusters 4–7 represent post-tunnel states.
- Breakout: Transitions from Breakout tunnel clusters into later clusters pass through a singular point, marking option termination after the left tunnel is carved.The transition structure is shown with states on the left and dynamics arrows over a 3D t-SNE map.
- Breakout: Cluster 7 represents an emulator bug separately and receives high value estimates, indicating representation based on game dynamics rather than pixels alone.The agent assigns this incident estimates similar to the other tunnel clusters.
- Breakout: Time-colored maps expose a local optimum in which the ball repeatedly bounces without hitting bricks, helping explain poor performance on Breakout’s second screen.The repeated reflection pattern occurs when only a few bricks remain.
- SAMDP: SAMDP automatically recovers most of the manually identified policy hierarchy, with consistent clusters and sparse transitions.Mean cluster images further reveal the skills underlying the learned policy.
4.4 Seaquest
Seaquest exposes a mismatch between immediate combat rewards and delayed diver rewards, and the learned representation reflects oxygen, diver status, and termination conditions. The resulting policy prioritizes enemy destruction and contains an imperfect refueling option.
- Environment: Seaquest is difficult for DQN because shooting enemies is immediately rewarded, whereas rescuing divers requires delayed, longer-horizon planning.Reward clipping further diminishes the bonus associated with collecting and rescuing six divers.
- Representation: The t-SNE clusters partition Seaquest states mainly by oxygen level and collected-diver status, with additional refueling and termination clusters.Low-oxygen clusters include flickering oxygen-bar states, while termination clusters reflect distinct visual end conditions.
- Representation: Saliency analysis shows that the network can notice a diver yet sometimes misinterpret it as an enemy and shoot at it.Other states show both diver and enemy detected, or the diver unnoticed.
- Policy: Value estimates correlate strongly with remaining oxygen and enemy count, but not with available or collected divers or proximity to refueling.The agent also never explores the screen bottom or collects more than two divers.
- Policy: The learned policy kills enemies while avoiding fire, ascends as oxygen falls, and avoids surfacing without a diver, but does not reliably collect divers.Without a collected diver, the ascent behavior is explicitly described as sub-optimal.
- Refuel option: Collecting a diver initiates a refuel option whose entry depends on oxygen level and whose intended termination is the fueling cluster.The option is imperfect because another refueling cluster does not exploit oxygen before ascending.
4.5 Pacman
Pacman’s learned representation organizes states by remaining bricks, value, spatial patterns, and agent location. The policy follows a specific sequence of bonus-brick collection, includes hiding and termination states, and reveals reward-clipping effects.
- Representation: Pacman clusters are partitioned by remaining bricks and value estimates, while also sharing brick patterns and agent locations.The analysis compares t-SNE maps colored by value and by the number of remaining bricks.
- Policy: The agent follows a location-based policy focused on collecting bonus bricks in a specific order while avoiding ghosts.The timing histogram supports the regularity of bonus-brick collection.
- Policy: The policy collects the bottom-left, top-right, bottom-right, and top-left bonus bricks through successive clusters.Clusters 1–4 cover the first two bonus-brick targets; later clusters cover the remaining two.
- Policy: Pacman’s representation includes termination clusters and a hiding cluster where the agent avoids ghosts in the top-left corner.Cluster 10 contains visible bonus-box states assigned lower values because of reward clipping.
- SAMDP: Sparse SAMDP transitions indicate that the agent remains for long periods in defined, spatially and temporally organized regions.The SAMDP visualization summarizes these cluster-level dynamics.
4.6 Enviroment modeling
The environment-modeling analysis checks how DQN handles initial and terminal states in its representation. Terminal states collapse into a singular zone, while initial states also form singular zones but receive incorrect value predictions.
- Environment modeling: Terminal states are mapped successfully into a singular zone across the analyzed games.The visualization focuses on termination and initial-state representations.
- Environment modeling: Initial states also map to singular zones, but their value predictions are wrong.This provides a direct diagnostic for the special treatment of padded initial states.
4.7 Score pixels
The paper investigates whether score pixels influence DQN predictions in Atari games, especially when repeated screens differ only by accumulated score. Saliency maps support the claim that DQNs base value estimates on these pixels.
- Repeated Atari screens can differ only by accumulated score, creating a risk that agents overfit score pixels and fail to generalize.
- Saliency maps indicate that DQNs use score pixels when forming value estimates across different games.
- The authors suggest training an agent without score pixels to further investigate their effect on network predictions.
4.8 SAMDPs
The authors evaluate whether SAMDP captures DQN value structure and policy behavior. Similar value estimates and positive policy–reward correlations indicate that the learned SAMDP model fits the data and is behaviorally consistent.
- SAMDP evaluation compares its value estimates with DQN estimates using VMSE and cluster-averaged DQN values because the true value function is unknown.The evaluation also examines the SAMDP value function and greedy policy.
- DQN values and SAMDP values are very similar, indicating that the SAMDP model fits the data well.
- For most states, greedy-policy decisions have positive correlation with trajectory reward, supporting the consistency of the SAMDP model.
- A train–test evaluation compares the greedy policy transition matrix with matrices from highly and poorly rewarded test trajectories.The comparison varies the number of top- and bottom-rewarded trajectories used.
5. Conclusions
The paper uses learned DQN features and SAMDP analysis to expose hierarchical state structure, interpret policies, and identify ways to improve agent behavior. It also argues that this methodology can support debugging and policy improvement without retraining.
- DQN features map states into sub-manifolds whose cluster dynamics reveal hierarchical structures and options with defined initiation and termination rules.
- Processing neural activity allows the authors to describe learned DRL policies with simple logic rules.
- The methodology aims to gray the black box by helping explain why DQNs work and interpret learned policies when poor performance is difficult to diagnose.
- SAMDP helped improve the existing policy in the Eject experiment without retraining it.
6. Appendix
The appendix visualizes Atari state representations with t-SNE maps colored by time, value, and game-specific measures. These views expose repeated states, trajectories, and partitions associated with gameplay variables.
- A Breakout t-SNE map colored by time highlights states that the agent revisits repeatedly.
- A second Breakout t-SNE visualization colors states by value and shows an example trajectory through the map.
- Seaquest t-SNE states are partitioned by divers collected, sea-level proximity, diving depth, and oxygen level.