PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning

Akanksha Atrey, Kaleigh Clary, David Jensen

arXiv:1912.05743v2cs.LGcs.AI

TL;DR

Whether saliency maps provide useful explanations of deep RL behavior has not been empirically established. The paper surveys their use and tests map-derived hypotheses with counterfactual interventions in Atari environments. It concludes that saliency maps are better treated as exploratory than explanatory tools.

  • Problem

    Saliency maps are used to explain deep RL behavior, but their explanatory validity has not been empirically evaluated under controlled conditions.

    Source: §1 · p. 1, §4 · p. 5

  • Method

    The authors survey saliency-map explanations and use TOYBOX counterfactual interventions on Atari game state to evaluate map-derived hypotheses.

    Source: §1 · p. 1, §4 · p. 6

  • Results

    Saliency maps cannot be trusted to reflect causal relationships between semantic concepts and agent behavior.

    Source: §6 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    Saliency maps are best used as exploratory rather than explanatory tools, with other methods or experimental evaluation providing support.

    Source: §6 · p. 9, §7 · p. 10

    The evaluation uses TOYBOX, a capable simulation environment; evaluation in non-intervenable environments may be possible but more tedious.

    Source: §6 · p. 10

Abstract

from arXiv · show

Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, a common benchmark for deep RL, to evaluate three types of saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.

1 INTRODUCTION

Saliency maps are widely used to explain deep RL behavior, but their explanatory value has not been empirically evaluated as it has for image classifiers. This work surveys those uses and proposes counterfactual evaluation of the hypotheses drawn from maps.

1 INTRODUCTION

  • Deep RL saliency is used to interpret internal representations and behavior across frames, unlike image-classification saliency focused on pixel importance for a label.Sequential actions can have effects that interact over long periods.Source: §1 · p. 1
  • The study identifies a gap in empirical evaluation of saliency maps as explanations of deep RL behavior.Prior evaluations addressed image classifiers, but the corresponding literature for RL agents was absent.Source: §1 · p. 1
  • The authors survey saliency-map explanations, introduce an interventional evaluation method, and test whether pixel-level inferences correspond to human semantic-level inferences.In the Breakout example, the saliency pattern around the tunnel is not symmetric under reflection interventions.Source: §1 · p. 1, Figure 1 · p. 2

2 INTERPRETING SALIENCY MAPS IN DEEP RL

A Breakout saliency pattern that appears to highlight a brick tunnel does not persist under reflections. The section explains why pixel interventions can diverge from game semantics and distinguishes intervention types for testing behavior hypotheses.

2 INTERPRETING SALIENCY MAPS IN DEEP RL

  • Reflections of the Breakout scene do not preserve the tunnel’s saliency pattern, so its apparent prominence does not establish that the agent aims at tunnels.Neither the tunnel alone nor its relative position to the ball and paddle accounts for the original saliency intensity.Source: §2 · p. 2

2.1 SALIENCY MAPS AS INTERVENTIONS

  • Saliency maps intervene on pixels and compare the network’s logits before and after image changes, but these changes are not naturalistic and may produce inputs where the learned function is not well-defined.Methods include changing pixel color, adding a gray mask, blurring a region, or masking objects with the background color.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 EXPLANATIONS FROM SALIENCY MAPS

  • Pixel changes need not alter semantic concepts or game state, so semantics-preserving interventions can test behavior hypotheses more precisely.The section distinguishes distortion, semantics-preserving, and fat-hand interventions; fat-hand interventions also alter other relevant aspects of the system.Source: §2.2 · p. 3, Table 1 · p. 4

3 SURVEY OF USAGE OF SALIENCY MAPS IN DEEP RL LITERATURE

The survey finds that deep RL researchers use saliency maps to infer agent focus and learned policy features, while only a small share of claims include additional experimental evidence. It also identifies subjectivity, unfalsifiability, and expectation-driven interpretation as pitfalls.

3 SURVEY OF USAGE OF SALIENCY MAPS IN DEEP RL LITERATURE

  • From 90 papers, the survey identified 46 claims from 11 papers that used saliency maps as evidence about agent behavior.Source: §3 · p. 4

3.1 SURVEY RESULTS

  • All surveyed claims treat salient regions as a proxy for agent focus, and 87% infer learned policy features by reasoning backward from saliency and behavior.Source: §3.1 · p. 4
  • Only 7% (3 out of 46) of surveyed saliency-based claims include additional or more direct experimental evidence.Two of the three attempts manipulate semantics in pixel inputs to assess agent responses.Source: §3.1 · p. 5

3.2 COMMON PITFALLS IN CURRENT USAGE

  • Saliency methods do not formalize abstract concepts such as “aiming” or quantitatively compare semantically meaningful behavioral consequences, contributing to subjective conclusions.Source: §3.2 · p. 5
  • Some interpretations are unfalsifiable because abstract claims such as a diver being “misunderstood” lack a clear conceivable test.Source: §3.2 · p. 5
  • Researchers’ expectations can bias both whether semantic mappings are inferred and the nature of those mappings.Source: §3.2 · p. 5

4 METHODOLOGY

The methodology addresses the lack of controlled empirical evaluation of saliency-based inferences by turning them into falsifiable hypotheses and testing them with interventions on game state. It uses counterfactual semantic conditions to assess the relationship between saliency and learned representations.

4 METHODOLOGY

  • The method targets the lack of controlled evaluation by intervening on game state to create counterfactual semantic conditions and assess saliency against learned representations.Source: §4 · p. 5
  • The proposed claim pattern links a salient concept set to a learned representation and resulting behavior, making hypotheses amenable to empirical analysis.The Breakout example reformulates tunnel-aiming as a claim about salient bricks, recognizing a partially complete tunnel, and steering the paddle.Source: §4 · p. 6
  • Counterfactual evaluation relies on a static learned representation whose relationship with saliency should remain invariant under pixel-input manipulations.Source: §4 · p. 6
  • The method intervenes directly on game state rather than pixels, leaving the image-generating process intact, and uses TOYBOX Atari implementations to generate interventional data.Saliency on an object is defined as average saliency over a bounding box around it.Source: §4 · p. 6

5 EVALUATION OF HYPOTHESES ON AGENT BEHAVIOR

Three Atari case studies test hypotheses drawn from saliency maps using counterfactual interventions and observational analysis. Across Breakout and Amidar, the findings show that saliency patterns do not reliably establish the proposed relationships between semantic features and agent behavior.

Evaluation setup

  • The case studies used TOYBOX counterfactual states to evaluate hypotheses with Jacobian, perturbation, and object saliency maps in deterministic Breakout and Amidar environments.The agents were trained with A2C using a CNN-based implementation and default hyperparameters; the authors describe the model choice as arbitrary.Source: §5 · p. 6

Case Study 1: Breakout Brick Translation

  • In Breakout, saliency on the tunnel changed under horizontal brick translations rather than remaining invariant, with location potentially confounding tunnel formation and salience.The authors also shifted the ball and paddle left in a reflection intervention.Source: §5 · p. 7, Figure 3 · p. 7

Case Study 2: Amidar Score

  • Different Amidar score interventions produced differing performance degradation but similar saliency patterns, leaving behavior as measured by reward underdetermined by salience.The reported reward–saliency correlations ranged from 0.041 to 0.274 and were insignificant for all but one intervention.Source: Figure 4 · p. 8, §5 · p. 8

Case Study 3: Amidar Enemy Distance

  • Amidar enemy distance showed no observational correlation with enemy saliency, and intervening on enemy position produced an effect close to zero.The Jacobian and perturbation methods likewise showed no correlation in the distance experiments.Source: §5 · p. 8, Figure 5 · p. 9

Cross-case conclusions

  • Observational analysis can expose spurious correlations without intervention, while intervention can help identify common causes behind high correlations.The authors caution that temporal association, such as tunnel formation followed by higher saliency, does not generally imply causal dependence.Source: §5 · pp. 7–9

6 DISCUSSION AND RELATED WORK

The discussion argues that counterfactual evaluation can expose mistaken inferences from saliency maps and help make explanations more falsifiable. It also describes the approach’s potential reach and boundaries, alongside related explanation methods.

Discussion

  • Counterfactual evaluation highlights risks of inferring association from observation, overlooking contradictory experimental evidence, and missing confounding in temporal processes.The paper concludes that saliency maps alone cannot reliably explain deep RL behavior and should be combined with supporting explanation methods or experimental evaluation.Source: §6 · p. 9
  • Separating semantic concepts X, learned representations R, and observed behavior B can help design more specific, falsifiable explanations.The paper notes that an agent’s semantic space may differ from the environment’s latent-factor space.Source: §6 · pp. 9–10

Generalization and limitations

  • The graphical-model framework can extend to other image-input deep RL domains, while the authors note evaluation may be more tedious in non-intervenable environments.They suggest pixel-input manipulation as one way to reproduce some interventions in observation-only environments.Source: §6 · p. 10
  • The methodology analyzes feed-forward agents and may carry over to recurrent agents, but repeated interventions may work poorly with recurrent agents’ capacity for memorization.Source: §6 · p. 10

Related work

  • Related deep RL explanation approaches include counterfactual-state generation, reward decomposition, and domain-specific languages for human-readable policies.The discussion also cites TOYBOX as a system for experimental evaluation and behavioral tests.Source: §6 · p. 10
  • Prior deep-network work has evaluated saliency methods for image classification and examined their robustness or theoretical basis.The cited examples include input-variance tests and evaluations of existing saliency methods.Source: §6 · p. 10

7 CONCLUSIONS

The paper surveys saliency-map explanations, proposes an evaluation methodology, and compares pixel-level inferences with semantic-level interventions. Its conclusion is that saliency maps are exploratory rather than explanatory tools for deep RL.

Conclusions

  • The paper surveys how saliency maps have been used, proposes an evaluation methodology, and examines whether learned representations can be inferred from the maps.Source: §7 · p. 10
  • The evaluation compares pixel-level inferences from saliency maps with semantic concept-level inferences tested through human-level interventions.Source: §7 · p. 10
  • The results indicate that saliency maps cannot be trusted to reflect causal relationships between semantic concepts and agent behavior, so the authors recommend them as exploratory rather than explanatory tools.Source: §7 · p. 10

A SALIENCY METHODS

The appendix illustrates three saliency methods evaluated for Amidar: perturbation, object, and Jacobian saliency.

Saliency methods

B MODEL

This block describes the A2C Atari setup, surveys published saliency-based behavioral claims, and extends a causal graphical model to image-input RL.

B MODEL

  • The agents used OpenAI Baselines A2C with the Mnih CNN architecture and were trained for 40 million iterations using RMSProp defaults.The hyperparameters are listed in Table 4.Source: B MODEL · p. 14, Table 4 · p. 14

C SURVEY OF USAGE OF SALIENCY MAPS IN DEEP RL LITERATURE

D GENERALIZATION OF CAUSAL GRAPHICAL MODEL

  • For image-input RL, the causal graphical model extends to include underlying game-state variables; Breakout’s A2C logits are split into action logits and value.Figure 7 depicts the extension for Breakout.Source: D GENERALIZATION OF CAUSAL GRAPHICAL… · p. 20, Figure 7 · p. 20
  • Displayed-score interventions in Amidar produced differing reward degradation but similar perturbation saliency, indicating that rewards were underdetermined by salience.Table 6 reports small correlations between reward and saliency differences for the interventions.Source: Figure 8 · p. 21, Table 6 · p. 21

E EVALUATION OF HYPOTHESES ON AGENT BEHAVIOR

The Amidar score case study intervenes on displayed score and examines reward and saliency differences across three saliency methods.

E EVALUATION OF HYPOTHESES ON AGENT BEHAVIOR

  • The study evaluated displayed-score interventions in Amidar using perturbation and Jacobian saliency maps, with results shown in Figures 8 and 9.Source: §E.1 · p. 21
  • For all three methods, the study evaluated Pearson’s correlation between reward and saliency differences relative to the original trajectory; the reported results support the correlation plots in Figures 4c, 8c, and 9c.Source: §E.1 · p. 21
  • Pearson’s correlation was evaluated between reward and saliency differences from the original trajectory for object, perturbation, and Jacobian methods.Source: §E.1 · p. 21

E.2 CASE STUDY 3: AMIDAR ENEMY DISTANCE

In Amidar, interventions on enemy position and regression analyses found little evidence that player-enemy distance relates to saliency.

E.2 CASE STUDY 3: AMIDAR ENEMY DISTANCE

  • Enemy 1 was more salient throughout the game than the other four enemies, yielding a larger interventional sample size.Source: §E.2 · p. 21
  • Figures 10–11 show no substantial correlation or causal dependence between enemy distance-to-player and perturbation or Jacobian saliency; Jacobian performed worst.Source: §E.2 · p. 21, Figure 10 · p. 22, Figure 11 · p. 22
  • Tables 7–8 report very small regression effects and correlations, supporting a lack of correlation between observational and interventional distributions.Source: §E.2 · p. 21, Table 7 · p. 23, Table 8 · p. 23

Related papers

Loading 1912.05743v2…