Source-linked AI summary
A Survey on Interpretable Reinforcement Learning
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, Wulong Liu
TL;DR
High-stakes deployment needs RL policies that can be inspected, but deep RL remains difficult to understand and verify. This survey distinguishes model interpretability from post-hoc explainability and synthesizes interpretable RL across inputs, models, and decision-making. It concludes that these components provide a broad framework for organizing existing work while exposing trade-offs and scope boundaries.
Problem
Deep RL is not yet mature enough for high-stakes applications because its learned policies can be difficult to understand and inspect.
Method
The survey distinguishes interpretability from explainability and organizes recent RL research around interpretable inputs, transition or reward models, and decision-making.
Results
The survey provides a comprehensive view of interpretable RL spanning structured and symbolic approaches, interpretable models, and interpretable decision-making.
Takeaways & Limitations
Interpretable RL should be considered across multiple components of the RL process rather than decision-making alone.
Takeaways & Limitations
The survey focuses on the intelligibility of trained models and leaves finer definitions of interpretability and explainability outside its scope.
Abstract
from arXiv · showhide
Although deep reinforcement learning has become a promising machine learning approach for sequential decision-making problems, it is still not mature enough for high-stake domains such as autonomous driving or medical applications. In such contexts, a learned policy needs for instance to be interpretable, so that it can be inspected before any deployment (e.g., for safety and verifiability reasons). This survey provides an overview of various approaches to achieve higher interpretability in reinforcement learning (RL). To that aim, we distinguish interpretability (as a property of a model) and explainability (as a post-hoc operation, with the intervention of a proxy) and discuss them in the context of RL with an emphasis on the former notion. In particular, we argue that interpretable RL may embrace different facets: interpretable inputs, interpretable (transition/reward) models, and interpretable decision-making. Based on this scheme, we summarize and analyze recent work related to interpretable RL with an emphasis on papers published in the past 10 years. We also discuss briefly some related research areas and point to some potential promising research directions.
1 Introduction
Deep RL enables powerful sequential decision-making but remains difficult to understand, train, generalize, and verify for high-stakes deployment. This survey therefore presents a broader account of interpretable RL spanning inputs, models, and decision-making.
- Deep RL combines neural-network approximation with sequential decision-making to achieve successes in games and robotics.
- Learned DRL policies are hard to understand because deep neural networks are black boxes.
- DRL policies can require substantial data, be difficult to train and replicate, overfit training environments, and generalize poorly.
- These weaknesses contribute to safety and adversarial vulnerabilities, leaving DRL not ready for high-stakes applications such as autonomous driving or healthcare.
- Interpretable policies support ethical, legal, operational, and usability concerns by making decisions inspectable before deployment.
- The survey distinguishes interpretability from explainability and organizes interpretable RL around inputs, transition or reward models, and decision-making.
2 Background
RL models agents as repeatedly interacting with environments, while deep RL uses neural networks and distinct value-based or policy-gradient methods. Hierarchical RL adds temporal and hierarchical abstractions to complex decision-making.
- An RL agent repeatedly receives observations, chooses actions, and receives new observations and rewards through an interaction loop.
- RL may additionally involve perception, learning from experience, knowledge representation and reasoning, or planning when the environment is modeled.
- RL is commonly formulated as an MDP or POMDP with states, actions, transitions, and rewards, while the agent learns a reward-maximizing policy.
- Deep RL uses neural networks to approximate policies or value functions and includes value-based methods and policy-gradient methods such as actor-critic algorithms.
- Hierarchical RL uses multiple decision levels, with higher-level controllers selecting temporally extended macro-actions for lower-level controllers.
3 Interpretability and Explainability
The survey distinguishes interpretability as a model property from explainability as an external, usually post-hoc operation. In RL, it applies interpretability across inputs, transition or preference models, and decision-making, while noting that global interpretability can be difficult and category boundaries overlap.
- Definitions: The survey uses interpretability for a model’s passive intelligibility and explainability for an active, usually post-hoc external methodology or proxy.
- Definitions: Interpretability may concern simulatability, decomposability, or algorithmic transparency, rather than a single universal definition.
- Definitions: The review focuses on interpretability and does not attempt finer definitions of the terminology beyond its survey scope.
- Interpretability in RL: In RL, interpretable processing includes the agent’s inputs, transition and preference models, and decision-making model.
- Interpretability in RL: Applying interpretability globally in RL requires simple components across inputs, preferences, transitions, and decision-making, which can be difficult in practice.
- Interpretability in RL: The survey organizes RL approaches into interpretable inputs, transition and preference models, and decision-making, while acknowledging that category boundaries are not sharp.
4 Interpretable Inputs
Interpretable inputs provide intelligible structure for RL agents, spanning pre-given relational representations, learned symbolic abstractions, and relation-centric representations. These inputs can support learning, generalization, transfer, reasoning, and planning, but current approaches often depend on hand-designed structures or struggle with complex real-world observations.
- Overview: Interpretable inputs include observations plus relational or hierarchical information that makes later decision-making easier to understand.The survey distinguishes pre-given structured representations from structures extracted from high-dimensional observations and additional interpretable knowledge.
- Structured Approaches: Structured and relational RL represents environments through objects, relations, predicates, and functions, enabling agents to learn and reason about objects.These approaches define task, reward, transitions, policies, or value functions over objects and their interactions, often using first-order logic.
- Structured Approaches: Explicit symbolic structures may improve generalization and transfer, but often require hand-designed representations, rely on non-differentiable operations, and are poorly suited to noisy complex environments.The survey identifies limited flexibility across task or input variations as an additional drawback.
- Learning Symbolic Representations: End-to-end approaches addressing symbol grounding connect learned symbolic representations with control and high-level planning.The survey presents this as a promising direction for learning abstractions rather than relying entirely on pre-given schemes.
- Relational Representations: Graph-based relation-centric representations can be fed to policy or value networks and used in hierarchical decision-making.They are presented as a way to compute interactions between entities for more interpretable inputs.
- Learning Symbolic Representations: Learning symbolic representations from raw data reduces state-space complexity through abstraction and can avoid hand-designing the symbolic domain, potentially improving adaptability across environments.The resulting modules can be used as preprocessing or trained end-to-end with object-oriented or relational RL.
5 Interpretable Transition and Preference Models
The survey organizes interpretable transition and preference models as environment or task representations that support human understanding and, in some cases, more data-efficient learning and transfer. It covers probabilistic, deterministic, neural, object-centric, and reward- or logic-based approaches, while noting a transparency–fidelity trade-off in neural models.
- Model-based interpretability: Interpretable transition and preference models help agents reason about decisions while also supporting human understanding and post-hoc explanation.Transition models represent environment dynamics; preference models represent task or reward-related information.
- Transition models: Interpretable transition models can reveal problem structure and decomposition, enabling more data-efficient exploration, generalization, and transfer learning.The survey discusses decision trees, graphical models, physics-based models, graph models, and neural networks with architectural inductive biases.
- Transition models: Probabilistic and deterministic approaches represent interpretable dynamics using decision trees, graphical or relational models, object-oriented representations, physics engines, and planning graphs.Examples include χ2-based structural tests, Bayesian physics models, and graph-search methods such as Dijkstra and A*.
- Neural and object-centric models: Neural transition-model approaches process high-dimensional inputs and increasingly use object-centric decompositions, but neural networks can reduce intelligibility despite high-fidelity simulation.Object-level methods segment and track entities or model local interactions to support compact dynamics and generalization across configurations.
- Preference and task models: The survey also covers interpretable reward and task representations, including deep decision-tree rewards, temporal-logic specifications, and learned relational task descriptions.These approaches address the difficulty of specifying rewards and the risk that careless reward engineering produces undesired behavior or value misalignment.
6 Interpretable Decision-Making
The survey groups interpretable decision-making into direct, indirect, architectural, and regularization-based approaches. These methods use representations such as decision trees, formulas, logic rules, programs, and neural-symbolic architectures, but involve trade-offs in interpretability, expressivity, scalability, and performance.
- Interpretable policies can be learned directly, indirectly, through architectural choices, or through regularization.Direct methods search an interpretable policy space, whereas indirect methods first learn an effective policy and then imitate it with an interpretable model.
- Decision Trees: Decision trees represent policies or value functions through feature-based paths and interpretable leaves, but can be computationally expensive to learn.Variants include trees with parameterized decision nodes, linear models at leaves, or parametric policies attached to leaves.
- Direct Approaches: Direct policy representations include fuzzy rules, first-order logic, programs, formulas, and dedicated polynomial functions.These approaches differ in how they constrain policy structure, search the representation, and balance expressivity with tractability.
- Programs and Formulas: Programmatic and formula-based policies make decision procedures explicit by expressing them as executable programs or closed-form mathematical functions.Formula-based methods restrict available operators or formula depth, while programmatic methods project continuous policy updates into a programmatic policy space.
- Indirect Approach: Indirect approaches separate reinforcement learning from interpretability by distilling or imitating an already-trained expert policy with an interpretable one.This setting can also exploit access to the expert policy’s value function and uses supervised learning after the RL problem has been solved.
7 Explainable RL
Explainable RL usually provides post-hoc, often model-agnostic explanations of decisions made by a trained agent, whereas interpretable-policy approaches change the controller itself. Explanations can be visual, textual, causal, or preference-based, but their validity and evaluation remain important concerns.
- Explainable RL commonly uses post-hoc, often model-agnostic procedures to provide functional explanations of black-box agent decisions.The target may be features, predicates, outcomes, causal relations, or other information associated with an action.
- Visual Explanation: Visual explanations include t-SNE representations, saliency maps, perturbation-based maps, and visual summaries of agent histories.These methods analyze network activations, input sensitivity, or behavioral histories to explain decisions.
- Textual Explanation: Textual explanations identify state or outcome predicates associated with actions, and may reuse instructions supplied by a human tutor.Some approaches produce contrastive explanations using both state and outcome predicates.
- Causal Explanation: Causal explanations learn or use causal structure to explain action choices and possible consequences through contrastive reasoning or opportunity chains.Opportunity chains include information about what an action enables or causes.
- Interpretable Policy: A policy learned to explain a black-box controller differs from an interpretable policy used as the proper controller.The former is a post-hoc explanatory proxy, while the latter is the decision-making policy itself.
- Limitations: Post-hoc explanations may appear plausible yet fail to reflect the model’s actual decision process, whereas this issue does not arise with an interpretable policy.The distinction matters when the goal is understanding decision-making rather than merely justifying an action to a user.
8 Open Problems and Research Directions
The survey identifies full interpretability, performance, scalability, and evaluation as open problems for interpretable RL. In particular, high-dimensional inputs and computationally difficult interpretable policy learning make complete transparency difficult to achieve.
- Full Interpretability in RL: Few deep RL approaches that accept high-dimensional inputs achieve full interpretability across inputs, models, and decision-making.The survey presents full interpretability as a difficult open problem rather than an established capability.
- Interpretability vs Performance: The relationship between interpretability and final performance remains unresolved because transparent models may be less capable, while simple models can perform well in some domains.The survey notes competing views about whether deep-learning black boxes are always necessary.
- Interpretability vs Scalability: Learning a fully interpretable policy may be computationally costly because decision-making can resemble program synthesis, an NP-hard problem.This creates a potential trade-off between the desired degree of interpretability and computational scalability.
- Evaluation: There is no common ground yet for evaluating interpretability and explainability because these concepts serve different purposes and have multiple meanings.The survey therefore identifies evaluation metrics as a central unresolved question.
9 Conclusion
The survey synthesizes interpretability across RL inputs, models, and policies, while also reviewing post-hoc explainability and open problems. It emphasizes that interpretable RL is increasingly important for responsible deployment, although complete transparency may require selectively prioritizing higher-level components.
- The survey organizes interpretable RL around interpretable inputs, transition and preference models, and policies, with decision-making treated as especially critical.It also includes a short review of post-hoc explainability and identifies future research directions.
- As RL-based systems become more widespread, interpretability concerns become increasingly pressing for responsible deployment.The survey links this concern to the broader ethical implications of algorithmic and automated systems.
- Fully intelligible RL may be difficult to achieve, motivating hierarchical designs that prioritize interpretability at higher levels such as subgoals.Lower-level components may remain partly opaque while higher-level behavior is made more transparent.
- Algorithmic intelligibility is presented as a promising step toward greater accountability and more trustworthy AI, but it does not by itself reveal all social implications.Understanding broader impacts requires examining the surrounding system rather than only the model’s internal operation.