PaperlayerView exact revision on arXiv

Source-linked AI summary

Transparency and Explanation in Deep Reinforcement Learning Neural Networks

Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, Katia Sycara

arXiv:1809.06061v1cs.LGstat.ML

TL;DR

Deep reinforcement learning networks are opaque, limiting humans’ ability to understand agent behavior and decisions. The paper integrates object recognition into deep RL to produce object saliency maps, evaluates them in Ms. Pacman and human experiments, and reports measurable human matching performance.

Abstract

from arXiv · show

Autonomous AI systems will be entering human society in the near future to provide services and work alongside humans. For those systems to be accepted and trusted, the users should be able to understand the reasoning process of the system, i.e. the system should be transparent. System transparency enables humans to form coherent explanations of the system's decisions and actions. Transparency is important not only for user trust, but also for software debugging and certification. In recent years, Deep Neural Networks have made great advances in multiple application areas. However, deep neural networks are opaque. In this paper, we report on work in transparency in Deep Reinforcement Learning Networks (DRLN). Such networks have been extremely successful in accurately learning action control in image input domains, such as Atari games. In this paper, we propose a novel and general method that (a) incorporates explicit object recognition processing into deep reinforcement learning models, (b) forms the basis for the development of "object saliency maps", to provide visualization of internal states of DRLNs, thus enabling the formation of explanations and (c) can be incorporated in any existing deep reinforcement learning framework. We present computational results and human experiments to evaluate our approach.

Introduction

Autonomous agents need transparency so humans can understand their intent, performance, plans, and reasoning, while deep RL systems remain opaque. The paper proposes object-sensitive DRL and object-level saliency visualizations to make agent decisions intelligible.

  • Autonomous agents increasingly operate in human environments, making understandable behavior important for trust and appropriate human decisions.Source: Introduction · p. 1
  • Agent transparency concerns interfaces that help operators comprehend an agent’s intent, performance, future plans, and reasoning process.Source: Introduction · p. 1
  • Deep neural networks achieve strong performance in tasks including Atari control but cannot produce human-understandable accounts of their reasoning.Source: Introduction · p. 1
  • The proposed Object-sensitive Deep Reinforcement Learning model incorporates object features and object valence into existing DRL architectures.Source: Introduction · pp. 1–2
  • The paper evaluates object-sensitive learning, object-level visual explanations, and the approach through computational and human experiments.Source: Introduction · p. 2

Related Work

Related work covers deep RL architectures and visualization methods for black-box models, alongside object recognition techniques based on convolutional neural networks.

  • DQN combines Q-learning with deep neural networks, while DDQN reduces overestimation and DuelingDQN separates state and action values.Source: Related Work · p. 2
  • Prior computer-vision work visualizes neural models by maximizing neuron activity or approximately reconstructing layer inputs from outputs.Source: Related Work · p. 2
  • Object recognition identifies objects in images or video despite changes in size, scale, translation, and rotation.Source: Related Work · p. 2
  • Convolutional neural networks have substantially improved object-recognition performance.Source: Related Work · p. 2

Reinforcement Learning

Reinforcement learning models sequential interaction: an agent selects actions from states, receives rewards, and observes successor states. Deep networks can approximate action values across several RL paradigms.

  • Reinforcement learning solves sequential decision problems through experience as an agent repeatedly interacts with an environment.Source: Reinforcement Learning · p. 2
  • In Atari, the agent maps image states to control actions and receives a score-based reward plus the next image.Source: Reinforcement Learning · p. 2
  • The agent seeks actions that maximize rewards over time, using discounted return to trade off recent and future rewards.Source: Reinforcement Learning · p. 2
  • The action-value function can be computed recursively with dynamic programming.Source: Reinforcement Learning · p. 2
  • Policy-based methods model policies directly, whereas value-based methods estimate action values with function approximators such as deep neural networks.Source: Reinforcement Learning · p. 2

Object-sensitive Deep Reinforcement Learning

The object-sensitive DRL approach recognizes objects, encodes their types and locations in additional input channels, and feeds these channels with images into DRL networks. It supports existing frameworks and uses object-aware rewards to distinguish objects.

Object recognition

Object representation

Network architecture

Training representation

Evaluation

DQN Transparency via Object Saliency Maps

The paper develops object saliency maps that make deep reinforcement learning decisions more intelligible by identifying which objects influence an agent’s action and reward. Compared with pixel saliency, object-level visualizations clarify the attended objects and their relative importance.

Object-level explanations

  • Object saliency maps visualize which objects influence a reinforcement learning agent’s selected action and Q-value.The method ranks objects according to their effect on Q(s, a), supporting object-level explanations of decisions.Source: DQN Transparency via Object Saliency… · pp. 3–4

Pixel-level saliency

  • Pixel saliency ranks image pixels using the local derivative of Q(s, a), but pixel-level representations are not intelligible to people.The derivative indicates which pixels need to change least to affect the Q-value.Source: DQN Transparency via Object Saliency… · pp. 3–4

Visualization comparison

Object influence

  • Objects are evaluated by masking each one, recomputing Q-values, and using the Q-value difference to represent its influence.Positive influence corresponds to a “good” object associated with positive future reward; negative influence identifies a “bad” object under the paper’s interpretation.Source: DQN Transparency via Object Saliency… · p. 4

Computational cost

  • Computing an object saliency map requires 2k model forward passes when k objects are detected.The paper describes this cost as affordable because k is generally not too large and forward passes are fast during model testing.Source: DQN Transparency via Object Saliency… · p. 4

Human Experiments

Human experiments evaluated whether object saliency maps help people connect DRL behavior to game scenarios, explain decisions, and predict actions. Participants linked maps to scenarios, while prediction accuracy varied by trial and was broadly comparable to screenshots.

Matching Task

  • The matching task paired Pacman video clips with either matching or non-matching saliency-map frames, then elicited explanations for matched cases.The task used 20 test trials, evenly divided between matching and non-matching pairs.Source: Human Experiments · pp. 4–5

Prediction Task

  • In the prediction task, participants viewed either screenshots or object saliency maps before predicting Pacman’s next action and explaining their choice.The prediction task contained 10 test trials with participants assigned to one of the two conditions.Source: Human Experiments · p. 5

Matching Task

  • Participants matched object saliency maps to corresponding Pacman game scenarios with 61.0% average accuracy and AUC = 0.81.Accuracy improved from 56.5% in the first half to 65.5% in the last half of trials.Source: Human Experiments · p. 5

Prediction Task

  • 58.0%±12.8% accuracy for the object saliency map group versus 56.5%±10.4% for controls showed no significant overall difference in the difficult prediction task.Trial effects and trial-by-group interaction were significant; screenshots performed better on test 2, while saliency maps performed better on tests 3 and 9.Source: Human Experiments · p. 5

Discussion

  • Screenshots supplied object identities and environmental context, whereas saliency maps displayed object valence; the two representations produced equal overall prediction performance.Large trial differences motivated examining when each representation supports more accurate prediction.Source: Human Experiments · pp. 5–6

Prediction Task

  • Test 9 illustrated complementary interpretations: 60% using saliency maps predicted continuing downward toward pellets, whereas 75% using screenshots predicted turning left toward the cherry.The O-DDQN agent ultimately went down for the dots, and saliency maps highlighted the objects influencing that decision.Source: Figure 3 · p. 5, Human Experiments · p. 6

Discussion

  • Object-level visualizations may support debugging and testing when DRL agents produce unexpected or abnormal behavior.The paper also proposes examining learned policies to understand their benefits and improve decision making.Source: Human Experiments · p. 6

Limitations

  • The authors caution that insufficient training and group-testing limitations may have depressed saliency-map prediction performance.They suggest longer training and more comprehensible tutorials, while noting that performance depends strongly on situation.Source: Human Experiments · p. 6

Conclusion and Future Work

The paper integrates object recognition and object-based saliency maps into deep reinforcement learning, evaluating their visualization utility in Ms Pacman and human experiments. Future work targets natural-language explanations and realistic applications such as self-driving cars.

Related papers

Loading 1809.06061v1…