Source-linked AI summary
Explainability in Deep Reinforcement Learning
Alexandre Heuillet, Fabien Couthouis, Natalia Díaz-Rodríguez
TL;DR
Deep RL has advanced beyond classification into demanding applications, but its decisions remain difficult to explain and evaluate for diverse audiences. This survey reviews XRL methods, organizes them into transparent and post-hoc approaches, and examines their implications for training, debugging, communication, and deployment. It finds a broad but largely task- and algorithm-specific literature, motivating more global approaches and explanations for diverse stakeholders.
Problem
RL explainability remains insufficiently understood for complex models and diverse audiences, despite the need to justify decisions in critical applications.
Method
The paper reviews XRL studies and classifies explanations as transparent methods or post-hoc explainability, while examining their uses across RL settings.
Results
The reviewed literature contains numerous approaches, but most are designed for particular tasks or algorithms and do not generalize straightforwardly across applications.
Takeaways & Limitations
Future XRL should address more diverse audiences and develop actionable, comprehensive frameworks for explainable, trustable, and responsible RL deployment.
Takeaways & Limitations
Explanation evaluation remains difficult because definitions and metrics lack consensus, audiences differ, and explanation quality is subjective.
Abstract
from arXiv · showhide
A large set of the explainable Artificial Intelligence (XAI) literature is emerging on feature relevance techniques to explain a deep neural network (DNN) output or explaining models that ingest image source data. However, assessing how XAI techniques can help understand models beyond classification tasks, e.g. for reinforcement learning (RL), has not been extensively studied. We review recent works in the direction to attain Explainable Reinforcement Learning (XRL), a relatively new subfield of Explainable Artificial Intelligence, intended to be used in general public applications, with diverse audiences, requiring ethical, responsible and trustable algorithms. In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box. We evaluate mainly studies directly linking explainability to RL, and split these into two categories according to the way the explanations are generated: transparent algorithms and post-hoc explainaility. We also review the most prominent XAI works from the lenses of how they could potentially enlighten the further deployment of the latest advances in RL, in the demanding present and future of everyday problems.
1. Introduction
Deep RL has achieved strong results across complex applications, but its limited explainability creates barriers to acceptance, debugging, evaluation, and responsible deployment. This paper reviews XRL methods, categorizes them, and discusses how explanations should serve different audiences and goals.
- Motivation: Deep RL combines reinforcement learning with deep representation learning to solve optimization problems across high-dimensional, stochastic environments.Applications include games, robotics, resource management, traffic control, chemical optimization, and recommendation systems.
- Motivation: RL explainability is important for applications such as defense, finance, and medicine that require decisions and actions to be explained to human users.Explainability is also relevant for societal acceptance of promising RL applications.
- Motivation: Deep RL models are difficult to debug because behavior depends on the environment, reward design, observation encoding, model architecture, and training algorithm.An explainable model could help developers identify problems more quickly and accelerate RL-method development.
- Scope and contributions: The paper reviews recent XRL literature, proposes a categorization of methods, and discusses recommendations for future work.Its stated contributions include a state-of-the-art review, an attempt to categorize XRL methods, and discussion of future directions.
- Audience and evaluation: Explainability objectives include trustworthiness, causality, transferability, informativeness, fairness, confidence, accessibility, interactivity, and privacy awareness, with relevance varying by audience.Model understandability depends on both model transparency and users’ goals and cognitive skills.
- Audience and evaluation: Evaluating explanations is challenging because explainability lacks a uniform definition, audiences differ, and explanation quality is subjective.User studies can quantify perceived accuracy and clarity, while mental-model evaluations may ask users questions or predict an agent’s next action.
2. XAI in RL: State of the art and reviewed literature
The reviewed XRL literature covers task-specific explanations for different RL algorithms, tasks, audiences, and modalities. The paper organizes explainability approaches into transparent methods and post-hoc explanations, including image-, diagram-, and text-based outputs.
- Reviewed literature: The literature summary records each paper’s task, RL algorithm, explanation type, explanation level, and target audience.Explanation types include images, diagrams, and text; explanations may be local or global.
- Reviewed literature: The reviewed studies span games, navigation, instruction following, traffic, robotic manipulation, assembly, and collision avoidance across algorithms including DQN, DDPG, Q-learning, SARSA, PG, and A2C.The table includes both fundamental RL studies and deep RL applications.
- Taxonomy: XRL studies are classified into transparent methods and post-hoc explainability according to how explanations are generated.Post-hoc approaches include techniques such as SHAP and LIME applied after training.
2.1. Transparent algorithms
Transparent XRL methods make explanations part of the agent or expose interpretable internal representations, rather than applying explanation tools after training. Reviewed approaches include representation learning, symbolic abstractions, reward decomposition, attention, and causal influence models.
- Transparent algorithms: Standard DRL algorithms are not transparent by nature, motivating architectures that learn explanations alongside policies or expose interpretable representations.Transparent algorithms are designed to be explainable without external processing, whereas standard DQN, PPO, DDPG, and A2C are not.
- Explanation through representation learning: State Representation Learning builds low-dimensional, meaningful state representations from high-dimensional observations to clarify what the agent interprets as relevant for action.Approaches include observation reconstruction, forward prediction, inverse modeling, and prior-knowledge constraints.
- Explanation through representation learning: Disentangled and symbolic representations separate semantic factors or incorporate background knowledge, supporting more interpretable features, generalization, and knowledge exploitation.The reviewed work includes variational autoencoder-based disentanglement, InfoGAN-style latent codes, and symbolic abstractions combined with deep RL.
- Simultaneous learning of the explanation and the policy: Attention-based graph representations explicitly compute entity relations and use attention to identify interactions relevant to states, actions, and policies.In Box-World, entities along the solution path serve as attention sources, with arrows indicating attended entities and transparency encoding attention weight.
- Simultaneous learning of the explanation and the policy: Reward decomposition represents rewards by meaningful types, allowing action preferences to be compared as trade-offs and supporting explanations such as Minimal Sufficient Explanations.MSX identifies a small set of important positive and negative reasons among decomposed rewards; HRA adds predicted nonterminal reward components.
- Simultaneous learning of the explanation and the policy: Causal action-influence graphs generate why and why-not explanations by encoding cause-effect relations between actions, states, and rewards.A user study found significantly better task prediction and explanation goodness than unexplained or relevant-variable video-game play, while trust did not significantly improve.
2.2. Post-Hoc explainability
Post-hoc XRL analyzes an RL agent after training and execution, commonly using saliency maps for visual inputs or interaction histories to produce human-facing explanations.
- Post-Hoc explainability: Post-hoc explainability analyzes an RL algorithm after training and execution to make a black-box model somewhat explainable.
- Explanation through saliency maps: Saliency methods highlight image regions containing information most relevant to an agent, producing visual maps that can be interpreted by non-experts.
- Explanation through saliency maps: A perturbation-based method for actor-critic Atari agents removes pixel information through Gaussian-blur interpolation to generate crisp saliency maps.
- Explanation through interaction data: Interaction-data approaches use introspection to identify meaningful events in an agent’s history and transform them into visual explanations such as highlight videos.
- Explanation through interaction data: In Frogger, no highlight summarization method suited every agent or scenario, and monotonous agents lacked enough interaction variety for pertinent explanations.
- Explanation through interaction data: Showing every interaction aspect can confuse users by juxtaposing an agent’s best and poorest performances.
2.3. Other concepts aiding XRL
The survey discusses complementary concepts that may aid XRL, including interpretable representations, relational reasoning, compositionality, imitation learning, and transparency tailored to users.
- Other concepts aiding XRL: Related XAI concepts include representation disentanglement, continual learning, relational reasoning, compositionality, imitation learning, and transparency.
- Representation learning and relational reasoning: Disentangled representations and relational graphs can make visual or object-based state representations more interpretable by exposing separate features or object relationships.
- Compositionality as a proxy tool to improve understandability: Compositionality decomposes complex tasks into simpler ones, enabling policy reuse, quicker initialization, and faster learning through combined skills.
- Compositionality as a proxy tool to improve understandability: Maximum-entropy RL methods produced composable policies demonstrated on a Sawyer robot combining obstacle avoidance and Lego-block stacking.
- Improving trust via imitation learning: Imitation learning uses human demonstrations to train agents, potentially improving behavioral understanding, trust, and textual or verbal explanations.
- Transparency: Transparency exposes decision-making for debugging, with information ranging from simple progress bars to complex debug logs according to user background.
3. Discussion
The discussion finds that XRL has many specialized approaches but lacks a generic method, while future progress requires broader techniques and explanations suited to end users.
- Discussion: Reviewed XRL methods are often task-specific, algorithm-specific, and dependent on assumptions about the MDP or environment, limiting generalization.
- Discussion: The diversity of actor-critic models, algorithms, and environments makes holistic or generic explainability difficult.
- Future directions: The research community would benefit from global-oriented approaches that do not focus on a particular task or algorithm, as in LIME or SHAP.
- Future directions: Representation-learning concepts such as Hindsight Experience Replay, Hierarchical RL, and self-attention may improve mathematical interpretability but generally lack concrete explanations for end users.
- Future directions: Deep RL explainability still needs progress on preserving invariants and addressing assumptions from disentangled representation learning.
4. Conclusion and Future Work
The review analyzes how XAI approaches can inform RL training, debugging, and communication, while identifying broader requirements for explainable, trustable, and responsible RL. It highlights future needs including human-involving RL, theoretical foundations, feature interactions, continuous interpretations, and diverse audiences.
- The review analyzes state-of-the-art RL and XAI approaches for training, debugging, and communicating with different stakeholder audiences.
- Future XRL should address human-involving RL, uncertainty-based handoffs, curriculum learning, quality diversity, and open-ended learning.
- More theoretical foundations are needed for explainable-by-design deep reinforcement learning, alongside post-hoc methods adapted to DRL requirements.
- Explanations should account for feature interactions because reviewed methods decompose predictions into additive components attributed to individual features.
- Future XAI tools should target developers, testers, end-users, and the general public to produce actionable explanations for responsible RL deployment.
5. Appendix
The appendix provides an acronym glossary covering reinforcement-learning algorithms, model architectures, learning paradigms, and explainability concepts.
- A2C, DDPG, IDDPG, MADDPG, PPO, SARSA, and COMA name reinforcement-learning algorithms or algorithm families.
- AI, ML, and XAI denote Artificial Intelligence, Machine Learning, and Explainable Artificial Intelligence.
- CNN, R-CNN, DQN, GAN, HMM, and VAE identify neural, probabilistic, generative, or value-learning model types.
- HRA and HRL refer to Hybrid Reward Architecture and Hierarchical Reinforcement Learning, respectively.
- MDP and POMDP identify Markov Decision Process and Partialy Observable Markov Decision Process, while SRL denotes State Representation Learning.
- RL denotes Reinforcement Learning, while DRL denotes Deep Reinforcement Learning and XRL denotes Explainable Reinforcement Learning.