Source-linked AI summary
Explainable Reinforcement Learning: A Survey
Erika Puiutta, Eric MSP Veith
TL;DR
The survey addresses the lack of an overview of Explainable Reinforcement Learning methods and the need to understand decisions from autonomous RL systems. It defines key terms, classifies and assesses selected XRL methods, and finds that post-hoc methods predominate while human-centered evaluation remains limited. The authors therefore emphasize adapting explanations to non-expert users through interdisciplinary work.
Problem
The survey addresses the lack of a single overview of Explainable Reinforcement Learning methods and the need to understand decisions made by autonomous RL systems.
Method
The paper defines XAI-related terms, classifies selected XRL methods, presents representative methods in detail, and critically evaluates them.
Results
Post-hoc interpretability models are more prevalent than intrinsic models, while only five of thirteen selected XRL methods evaluate non-expert user satisfaction or utility.
Takeaways & Limitations
XRL progress requires human-centered explanations and interdisciplinary work combining AI or ML with psychology, philosophy, and human-computer interaction.
Takeaways & Limitations
The surveyed methods often neglect the human recipient of explanations, and only three of thirteen selected methods combine contrastive explanations with explicit consideration of the human user.
Abstract
from arXiv · showhide
Explainable Artificial Intelligence (XAI), i.e., the development of more transparent and interpretable AI models, has gained increased traction over the last few years. This is due to the fact that, in conjunction with their growth into powerful and ubiquitous tools, AI models exhibit one detrimential characteristic: a performance-transparency trade-off. This describes the fact that the more complex a model's inner workings, the less clear it is how its predictions or decisions were achieved. But, especially considering Machine Learning (ML) methods like Reinforcement Learning (RL) where the system learns autonomously, the necessity to understand the underlying reasoning for their decisions becomes apparent. Since, to the best of our knowledge, there exists no single work offering an overview of Explainable Reinforcement Learning (XRL) methods, this survey attempts to address this gap. We give a short summary of the problem, a definition of important terms, and offer a classification and assessment of current XRL methods. We found that a) the majority of XRL methods function by mimicking and simplifying a complex model instead of designing an inherently simple one, and b) XRL (and XAI) methods often neglect to consider the human side of the equation, not taking into account research from related fields like psychology or philosophy. Thus, an interdisciplinary effort is needed to adapt the generated explanations to a (non-expert) human user in order to effectively progress in the field of XRL and XAI in general.
1 Introduction
The survey motivates explainable AI and XRL by highlighting the performance-transparency trade-off and the need to understand autonomous RL decisions. It introduces RL, defines interpretability, and reviews selected XRL methods to address the lack of an overview.
- Motivation: More powerful and flexible AI models become more opaque, creating a performance-transparency trade-off that motivates explainability.The literature uses related terms including readability-performance, accuracy-comprehensibility, and accuracy-interpretability trade-offs.
- Survey scope: The work reviews XAI terminology, classifies selected XRL methods, presents representative methods, and critically evaluates existing approaches.Its focus is XRL because the literature contains fewer overviews of XRL than of XAI generally.
- The importance of explainability: Explainability supports trust, acceptance, and the justification of AI decisions, including in legal and critical-infrastructure settings.The survey connects transparency with user trust and acceptance, GDPR-related explanation rights, and autonomous AI in critical infrastructures.
- Reinforcement Learning: Reinforcement Learning autonomously learns actions through trial and error, using rewards to learn a policy that maximizes cumulative reward.RL is commonly represented as a Markov Decision Process when the next state depends only on the current state and action; partially observable settings use POMDPs.
- Definition of important terms: The survey argues that interpretability requires both generating explanations and presenting them in ways understandable to human non-experts.It proposes using interpretability as a uniform term for explanations that ultimately enable users to predict model behaviour.
2 XAI Taxonomy
The taxonomy classifies interpretability methods by when explanations are extracted and by explanation scope. It distinguishes intrinsic from post-hoc approaches and global from local explanations, while also relating these categories to model specificity.
- Taxonomy: XAI methods are classified by extraction timing as intrinsic or post-hoc and by explanation scope as global or local.These two factors provide the survey’s primary taxonomy for organizing explainability methods.
- Global versus local: Global explanations describe general model behaviour, whereas local explanations explain a specific prediction or decision and feature contributions.Global methods inspect model structures; local methods ask why an output occurred for an instance or group of instances.
- Intrinsic versus post-hoc: Intrinsic models are designed to be interpretable during training by restricting complexity, while post-hoc methods analyze trained models using simpler explanatory models.Decision trees exemplify intrinsic interpretability, whereas surrogate models and saliency maps exemplify post-hoc interpretability.
- Trade-offs: Intrinsic models usually provide accurate explanations but can sacrifice prediction performance, while post-hoc methods generally preserve original accuracy but may yield less simple explanations.Post-hoc models can be applied to intrinsic models, but the reverse is not necessarily true.
- Model dependence: Model-specific techniques are limited to a model or model class, whereas model-agnostic techniques can be used with any model.Intrinsic methods are model-specific, while post-hoc methods are usually model-agnostic.
3 Non-exhaustive list of XRL methods
The survey classifies selected XRL methods by when explanations are generated and whether they are global or local, then examines representative approaches in detail. These methods include programmatic policies, hierarchical skill acquisition, and LMUT-based mimic learning, with reported gains in interpretability, generalization, or fidelity across settings.
- Survey selection: The literature search used Google Scholar, restricted papers to 2010–2020, and selected representative methods for detailed discussion.The authors chose the latest paper from each taxonomy quadrant, with one stated exception for Method C.
- Programmatic interpretability: PIRL mimics deep reinforcement-learning policies using a high-level, human-readable programming language instead of neural-network policy representations.The framework retains the goal of maximizing long-term reward while restricting the target policy space.
- Programmatic interpretability: Compared with DRL, NDPS achieved smoother driving, greater noise and sensor robustness, easier interpretation, and better generalization, despite slower lap times.The evaluation used the Open Racing Car Simulator, where the controller sets five driving parameters.
- Hierarchical skill acquisition: Hierarchical multi-task policies decompose complex tasks into simpler subtasks and can reuse learned base policies, requiring only weak human supervision to decide which skills to learn.A stochastic temporal grammar models temporal relationships and task priorities.
- Hierarchical skill acquisition: The hierarchical framework learned more efficiently than a flat policy, generalized well to new environments, and was inherently interpretable.The framework was trained and evaluated on object-manipulation tasks in Minecraft.
- LMUT mimic learning: LMUTs approximate a complex model’s Q-function while supporting feature-importance analysis, rule extraction, and super-pixel calculation.The framework generates mimic data either during experience training or through active play using the mature DRL agent.
- LMUT mimic learning: LMUT achieved higher fidelity with fewer leaves than CART and FIMT variants, the highest fidelity in Cart Pole, and the best overall play performance among compared models.M5 trees had higher performance in two other environments, although LMUT performance was comparable.
4 Discussion
The survey classifies and evaluates XRL methods, finding that post-hoc approaches dominate and that human-centered explanation design remains underdeveloped. It argues that future progress requires adapting explanations to users through interdisciplinary work.
- The survey organizes selected XRL methods by scope and information-extraction time, then examines representative methods from each category.The categories distinguish global versus local scope and intrinsic versus post-hoc extraction.
- Post-hoc interpretability is more prevalent because simplifying complex RL models appears easier than designing inherently simple ones.The survey associates this pattern with a performance-interpretability trade-off for explainability models themselves.
- Global and local methods are distributed roughly evenly, but human users may prefer policy-level explanations over explanations of individual actions.The cited study reports that human users tend to favor explanations about policy rather than single actions.
- Human audience considerations are often neglected, although explanations should adapt to users’ experience, education, culture, and domain knowledge.The survey also notes that contrastive explanations can be more intuitive and valuable for lay users.
- Only five of thirteen selected XRL methods evaluate non-expert satisfaction or utility, and only three of those provide contrastive explanations.The survey identifies Madumal et al., Sequeira and Gervasio, and van der Waa et al. among the methods combining these considerations.
- The survey concludes that interdisciplinary work across AI, psychology, philosophy, and human-computer interaction is needed for human-centered XRL.This recommendation is tied to developing explanations that support effective human-computer interaction and cooperation.