Source-linked AI summary
Explainable Reinforcement Learning Through a Causal Lens
Prashan Madumal, Tim Miller, Liz Sonenberg, Frank Vetere
TL;DR
The paper addresses how reinforcement-learning agents can provide causal explanations of their actions. It learns an action-influence structural causal model during reinforcement learning and uses counterfactual analysis to generate explanations. In a 120-participant Starcraft II study, the model performed significantly better than baselines on task prediction and explanation goodness, but not trust.
Problem
In model-free reinforcement learning, environment dynamics needed for counterfactual explanations are unknown, while causal explanation models for RL agents were absent.
Method
The approach extends structural causal models with action-dependent equations and learns approximate equations during RL using a given causal-direction DAG.
Results
The model performed significantly better than baselines on task prediction and explanation goodness, but its effect on trust was not statistically significant.
Takeaways & Limitations
Causal action-influence models can support counterfactual why and why-not explanations for model-free RL agents.
Takeaways & Limitations
The approach requires the causal model, specifically the causal-direction assumptions, to be given beforehand.
Abstract
from arXiv · showhide
Prevalent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen. In this paper, we use causal models to derive causal explanations of behaviour of reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigated: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results show that causal model explanations perform better on these measures compared to two other baseline explanation models.
Related Work
Prior work has explored policy and action explanations for MDPs and reinforcement-learning agents, but causal models for generating explanations in these settings remained absent.
- Earlier MDP work generated explanations by selecting relevant state variables or using sufficient and minimal domain-independent templates.
- Policy explanations have supported transparency, policy summaries, interactive instruction-based explanations, interpretable policies, and minimum sufficient explanations.
- Although causal explanation has received attention in explainable AI, causal explanation models for MDPs and reinforcement-learning agents had not generated explanations.
Causal Models for Explanations
The paper extends structural causal models with action-dependent effects to represent how reinforcement-learning actions influence environment variables and rewards.
- Starcraft II provides a partially observable real-time strategy setting with large state and action spaces; the implementation uses 4 actions and 9 state variables.
- Structural causal models represent worlds with exogenous and endogenous variables whose relationships are described by structural equations.
- Causal Models for Reinforcement Learning Agents: The action influence model adds actions to the SCM signature and uses action-specific structural equations to represent causal effects on variables.
- Causal Models for Reinforcement Learning Agents: Reward variables are sink nodes, defined as variables with no outgoing edges in the action influence graph.
- The model instantiates state variables using current MDP values, making exogenous variables irrelevant for that instantiation.
- Each state variable has one structural equation for each unique incoming action, such as An being influenced by S and B only under action Am.
Explanation Generation
Explanations are generated from learned action-influence causal chains, with minimally complete and contrastive forms designed to answer why and why-not action questions.
- Explanation generation has three phases: specify qualitative causal relationships, learn structural equations during reinforcement learning, and generate explanans from the SCM.
- The paper defines an explanation as an explanandum event paired with an explanans consisting of its selected causes, focusing on why and why-not action questions.
- A complete explanans follows an action’s causal chain to reward variables, including action-head and intermediate-node variables with their instantiated values.
- The approach assumes no knowledge of the explainee’s epistemic state and uses a simple heuristic for selecting intermediate explanatory nodes.
- Minimally Complete Explanations: The minimally complete explanation retains the action-head reason and immediate predecessor variables of reward nodes, omitting intermediate nodes to avoid overwhelming the explainee.
- ‘Why not?’ Questions: Contrastive explanations compare actual and counterfactual causal chains and retain only variables whose values differ between the two actions.
- Learning Structural Causal Equations: For model-free reinforcement learning, the approach assumes a given causal-direction DAG and learns approximate structural equations through multivariate regression during training.
Computational Evaluation
The evaluation measures whether action influence models support task prediction across six benchmark domains while tracking structural-equation training time. Results indicate generally reasonable prediction with negligible performance impact, although continuous actions remain a limitation.
- Results: The model achieved reasonable task prediction, supporting its potential usefulness for generating explanations.Domains with clearer causal structure, such as Starcraft, performed best in task prediction.
- Evaluation setup: Task prediction accuracy and structural-equation training time were evaluated across six reinforcement-learning benchmark domains using different algorithms.The evaluation used replay data and measured performance after training over 100 episodes.
- Model comparison: Linear regression was generally adequate for approximating the structural equations, while using multilayer perceptrons provided little additional benefit.The comparison concerns performance cost and approximation quality across the evaluated domains.
- Limitations: The model generally performed well with negligible performance impact, except that continuous actions in BipedalWalker were not handled accurately.The authors identify extending the model to continuous actions as future work.
Empirical Evaluation: Human Study
The human study evaluated whether causal-model explanations improve understanding and perceived explanation quality for an RL agent in StarCraft II, while also testing effects on trust. Across 120 participants, causal explanations improved task prediction and most explanation-quality measures, but trust differences were not statistically significant.
- Measures: Participants predicted the agent’s next action, rated explanation quality on a 5-point scale, and rated trust on a 5-point scale.Task prediction measured understanding; explanation quality assessed understanding, satisfaction, sufficient detail, and completeness; trust assessed confidence, predictability, reliability, and safety.
- Task Prediction: Task prediction scores differed significantly across explanation models, with means of 10.90 for causal, 10.20 for detailed causal, 8.97 for relevant-variable, and 8.53 for no explanations.The overall ANOVA reported p = 0.003; causal explanations significantly exceeded relevant-variable and no-explanation conditions in pairwise comparisons.
- Explanation Quality: Causal explanations significantly outperformed relevant-variable explanations on all explanation-quality metrics except Understand.The quality measures were understand, satisfying, sufficient detail, and complete, using aggregated video explanations.
- Trust: Trust differences across models were not statistically significant for confidence, predictability, reliability, or safety.Causal explanations had high means and medians, but the reported pairwise ANOVA results did not establish a significant trust effect.
- Scope and Limitation: The experiment assumed strong linearity for StarCraft II, allowing linear regression to learn structural causal models from only 9 state variables.This assumption enabled the selected regression approach in the study’s relatively small state representation.
Conclusion
The paper introduces action influence models that learn structural causal models during reinforcement learning and generate counterfactual explanations. Across benchmark and human evaluations, the approach performs significantly better on task prediction and explanation goodness, while requiring the causal model beforehand.
- Action influence models learn a structural causal model during reinforcement learning and generate why and why-not explanations through counterfactual analysis.
- The approach was evaluated on 6 benchmark RL domains for task prediction and in a human study with 120 participants.
- The model performed significantly better on task prediction and explanation goodness, but its impact on trust was not statistically significant.
- A central weakness is that the causal model must be given beforehand.