PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Deep Reinforcement Learning: State of the Art and Challenges

George A. Vouros

arXiv:2301.09937v1cs.LG

TL;DR

Explainability is needed for autonomous reinforcement-learning agents, especially in critical real-world settings, but the qualities of explainable and transparent deep RL methods remain insufficiently understood. The article reviews state-of-the-art methods and identifies components of explainable RL frameworks, while highlighting the need for comprehensive explanations.

  • Problem

    The literature lacks a concise understanding of the qualities of explainable and transparent deep reinforcement-learning methods for autonomous agents in critical real-world settings.

  • Method

    The article reviews state-of-the-art interpretability, explainability, and transparency methods for deep reinforcement learning and categorizes their explanation content and approaches.

  • Results

    The review identifies methods and explanation approaches spanning interpretable models and explanation content for deep reinforcement learning.

  • Takeaways & Limitations

    Explainable reinforcement-learning approaches should address combinations of explanation aspects to support comprehensive understanding by human operators.

  • Takeaways & Limitations

    The surveyed approaches do not yet address combinations of explanation aspects in as comprehensive a way as possible.

Abstract

from arXiv · show

Interpretability, explainability and transparency are key issues to introducing Artificial Intelligence methods in many critical domains: This is important due to ethical concerns and trust issues strongly connected to reliability, robustness, auditability and fairness, and has important consequences towards keeping the human in the loop in high levels of automation, especially in critical cases for decision making, where both (human and the machine) play important roles. While the research community has given much attention to explainability of closed (or black) prediction boxes, there are tremendous needs for explainability of closed-box methods that support agents to act autonomously in the real world. Reinforcement learning methods, and especially their deep versions, are such closed-box methods. In this article we aim to provide a review of state of the art methods for explainable deep reinforcement learning methods, taking also into account the needs of human operators - i.e., of those that take the actual and critical decisions in solving real-world problems. We provide a formal specification of the deep reinforcement learning explainability problems, and we identify the necessary components of a general explainable reinforcement learning framework. Based on these, we provide a comprehensive review of state of the art methods, categorizing them in classes according to the paradigm they follow, the interpretable models they use, and the surface representation of explanations provided. The article concludes identifying open questions and important challenges.

1 INTRODUCTION

Deep reinforcement learning systems increasingly act autonomously in complex, safety-critical settings, but their hidden reasoning mechanisms create unresolved explainability, accountability, and trust challenges. This survey reviews explainable DRL methods under a common framework and identifies open challenges.

  • Motivation: Deep knowledge encoded in hidden neural-network layers and value-function models has reduced the interpretability of autonomous systems for users, experts, engineers, and developers.This loss affects accountability, liability, safety, and trustworthiness as systems operate with higher levels of automation.
  • Human operators: Safety-critical autonomous applications require explanations that help human operators understand specific decisions, objectives, capabilities, and non-intuitive solutions.The paper connects this need to applications including autonomous vehicles and air traffic management.
  • Survey contribution: The article reviews state-of-the-art explainable DRL methods with emphasis on interpretability, explainability, transparency, and human-operator needs.It aims to assess capabilities and limitations, distinguish paradigms, describe interpretable models and explanation surfaces, and identify emerging challenges.
  • Challenges: DRL agents pose additional explainability challenges because they combine closed-box models with sequential decisions under partial observability, delayed rewards, dynamics, and multiple agents.These conditions complicate explanations of long-term reward-oriented reasoning and action selection.
  • Research questions: Open design questions include whether explainability should be added as a module or built into DRL systems from the outset, and what information should be communicated.The appropriate answer may vary across domains and deployment conditions.
  • Survey contribution: The survey organizes existing methods under a common framework, discusses their evaluation and transparency, and identifies important open questions and challenges.This framework is intended to provide a comprehensive account of explainable DRL approaches.

2 INTERPRETABILITY, EXPLAINABILITY AND TRANSPARENCY OF DRL METHODS

This section distinguishes interpretability, explainability, and transparency in DRL, then frames explainability around what an agent does, why it acts, and how explanations are communicated to people. It also formalizes the explanatory scope across models, responses, features, audiences, and levels of detail.

  • Scope: The section introduces DRL terminology and formalizes explainability problems for agents before presenting a general XDRL framework.It covers DRL preliminaries, conceptual distinctions, explainability desiderata, and formal problem definitions.
  • DRL preliminaries: A DRL agent learns a policy mapping states to actions or action distributions to maximize expected cumulative reward.Depending on the method, it may also model the environment, estimate state or state-action values, or learn a reward function.
  • Conceptual distinctions: Interpretability concerns explanation content from an interpretable model, whereas explainability concerns the surface representation of that content.Transparency additionally requires explanations understandable within the deployment context and its domain-specific constraints.
  • Explainability questions: XDRL must determine what qualitative understanding means, which responses and features to explain, how feature importance is assessed, and whose needs govern interpretation.The questions include individual responses versus aggregated trajectories, selected subsets of features, and the audience for explanations.
  • Qualitative understanding: Qualitative understanding explains policies, preferences, abilities, objectives, and action rationales in human-interpretable domain concepts rather than only numerical calculations.Such explanations can help operators infer broader behavioral patterns across circumstances, skills, and actions.
  • Interpretation and explanation: Interpretations may expose policy or objective elements through Q values, feature saliency, policy rollouts, MDP components, or raw model weights at different granularities.A function f(·) acts as a window into DRL models, while explanations select what information to communicate and how to present it.

3 FRAMEWORK AND PARADIGMS FOR XDRL METHODS

The XDRL framework organizes explanations around interpretable models and explanation functions connected to DRL policies, objectives, and responses. It distinguishes interpretable-box design from distillation and mimicking according to how interpretable models relate to the underlying DRL agent.

  • XDRL framework: A DRL agent typically combines a policy model with value, environment-dynamics, or reward models to select actions and optimize objectives.The policy maps state features to action probabilities, while reward and related models support policy tuning.
  • XDRL framework: The framework represents policy, objective, and response components through interpretable models that can operate locally, globally, or in combination.Explanation functions exploit these models, sometimes together with model-inspection outputs.
  • Interpretable-box design: Interpretable-box design replaces or supplements DRL components with interpretable models parameterized separately from the original DRL parameters.These interpretable models may be disconnected from constituent-model inspection facilities.
  • Distillation and mimicking: The mimicking and distillation paradigms train interpretable models using DRL behavior, but differ in their access to the agent’s internal models.Distillation transforms knowledge obtained through model-inspection facilities, whereas mimicking gathers interaction samples from the agent and environment.
  • Distillation and mimicking: Distillation directly uses constituent DRL models, while mimicking records decisions, state transitions, rewards, and assessed values during agent-environment interaction.The two paradigms are treated as distinct despite an acknowledged lack of a completely clear boundary in DRL.

4 XDRL METHODS: STATE OF THE ART

The review classifies XDRL approaches by their explanatory problem, interpretable representation, learning procedure, and access to DRL models or samples. Distillation and mimicking approaches emphasize high-fidelity interpretable models but vary in representation, splitting, construction, and sampling choices.

  • Review organization: The review describes each XDRL approach by its problem, paradigm, interpretable model, DRL-model access, evaluation objectives, methods, and results.It also summarizes applicability and online or offline information requirements in an appendix.
  • Review organization: Distillation and mimicking approaches primarily construct high-fidelity interpretable models relative to original DRL models.The approaches differ in several dimensions rather than forming a single uniform procedure.
  • Method dimensions: Reviewed methods vary in target representation, splitting rules, interpretable-model construction, and procedures for expanding training data.Examples include decision trees, logistic model trees, gradient boosting trees, LogiBoost, DecText, external classifiers, and different sampling schemes.
  • Scope: The review focuses on XDRL approaches that apply these interpretable-model methods to DRL agents rather than re-reviewing general deep-neural-network interpretation techniques.General methods are covered elsewhere, while their use within XDRL is examined here.

4.1 Solving the model inspection problem

Model-inspection methods explain DRL components through local or global feature attributions and interpretable surrogate models. The reviewed evidence highlights SHAP’s theoretical properties, sample efficiency, human-intuition consistency, and use in DRL action explanations.

  • LIME and local inspection: LIME learns a local interpretable model around a prediction and can explain individual DRL models or local action choices from state features.Its explanations use perturbed feature representations and interpretable models such as linear models or decision trees.
  • LIME and local inspection: SP-LIME selects representative, non-redundant individual explanations to provide a global perspective on model behavior.Simulated-user studies examined faithfulness, trust in predictions, and trust in the model; human studies examined classifier choice and feature removal.
  • Additive attribution: Additive feature-attribution methods are unified by theoretical results establishing a unique solution with local accuracy, missingness, and consistency properties.The reviewed work reports that several recent methods in the class lack one or more of these desirable properties.
  • SHAP: SHAP represents local explanations through Shapley values of a conditional-expectation function and relates feature contributions to model responses near an instance.Like LIME, it uses an interpretable linear model over binary instance representations.
  • SHAP: Kernel SHAP and Shapley sampling values learn accurate estimates with fewer original-model evaluations than comparison approaches.The cited evaluation identifies improved sample efficiency, while user studies report greater consistency with human intuition than DeepLIFT and LIME.
  • SHAP: SHAP explanations distinguished the target digit class from a near-class in MNIST convolutional-network classification.The comparison measured class probabilities and log-odds changes when image regions were masked.
  • DRL applications: In a traffic-signal DRL application, Kernel SHAP estimated how road-detector states affected the agent’s selected phases.The explanation related selected phases to traffic volumes and lane occupancy at a signalized roundabout.
  • Other inspection methods: LRP and NCA can inspect constituent DRL models by identifying feature importance and reducing predictor dimensionality for visualization and fast classification.These methods address model inspection rather than necessarily explaining the agent’s complete policy or outcomes.

4.2 Solving the policy explanation problem

Policy explanations in this section represent agent behaviour through selected trajectories, summaries, contrastive rollouts, and optional saliency maps. Human studies indicate that state selection and behavioural context often matter more than saliency alone.

  • Behavioural summaries: HIGHLIGHTS constructs behavioural summaries by selecting important states and neighbouring actions, while HIGHLIGHTS-DIV promotes diversity across state-space regions.Importance reflects the potential reward loss from taking an action not prescribed by the policy.
  • Human evaluation: HIGHLIGHTS summaries were evaluated in Pacman by asking participants to compare agents and rate summary helpfulness.The evaluation focused on observed behaviour rather than scores.
  • Human evaluation: Short summaries may not let participants align confidence with their ability to assess an agent’s abilities.Participants nevertheless preferred summaries for very well-trained agents with statistical significance.
  • Human evaluation: Behaviour summaries improved over baselines, and HIGHLIGHTS-DIV summaries improved over HIGHLIGHTS summaries.The reported comparisons concern generated summaries rather than policy scores.
  • Saliency and context: Adding saliency maps produced mixed results: state selection was more important, with no significant saliency-versus-nonsaliency differences in the study.Saliency maps were harder to interpret in videos than still images, but behavioural explanations placed highlighted features in context.
  • Behavioural summaries: No single summarization technique or interestingness element completely explains every agent across all task situations.Different combinations may be needed for agents with distinct capabilities and performance levels.
  • Contrastive explanations: Contrastive explanations compare policy rollouts with foil-policy rollouts to describe why the agent chose one action instead of another.Foil policies reflect user preferences and are contrasted with the agent’s policy through simulated rollouts.

4.2.3 Establishing appropriate trust via critical states.

Critical-state methods expose situations where action values differ sharply, enabling users to inspect possible errors and refine trust or decide when to intervene. Related SAMDP methods instead abstract learned DQN representations into spatio-temporal structures for policy inspection.

  • Critical states: A critical state is one where Q^π(s,a) is very high for some actions but mediocre or low for most alternatives.Users can inspect such states for false-positive or false-negative critical-state identifications.
  • Critical states: Inspecting critical states can help users refine trust and identify situations where they must take control.Incorrect actions in critical states indicate policy failure, while unexamined states may limit generalization about agent behaviour.
  • Critical states: The critical-state approach accesses Q-values without requiring an explicit interpretable model and supports user-in-the-loop deployment.Experts may assess false positives, false negatives, and incorrect actions before or during deployment.
  • Critical states: A key limitation is that users may struggle to generalize from presented critical states to states not identified as critical.Visualizing state-action pairs also becomes harder as state and action dimensionality increases.
  • SAMDP policy inspection: SAMDP analysis constructs spatio-temporal abstractions by selecting features, aggregating states, identifying skills, estimating dynamics, and choosing among candidate models.The resulting model supports analysis of DQN policies using state features, rewards, neural activations, and t-SNE visualizations.
  • SAMDP policy inspection: SAMDP visualizations, cluster analysis, and neural-activation tools aim to support understanding, debugging, and interpretation of DQN policy models.The method combines manually crafted features with abstractions learned from data.

4.2.5 Towards Better Interpretability in Deep Q-Networks.

The reviewed approaches improve interpretability in deep Q-learning through attention-based representations, influence analysis, and attention visualizations for visual agents. Their evaluations also expose limits involving shallow features, low action agreement, mixed saliency value, and incomplete interpretability assessment.

  • Attention-based Q-learning: Attention-based Q-learning associates state embeddings with action–Q-value key exemplars and visualizes those associations through clusters and reconstructed inputs.The model combines Bellman, reconstruction, and diversity losses during training.
  • Attention-based Q-learning: Directed exploration achieved training rewards comparable to state-of-the-art deep Q-learning models.The approach also provided insight into features extracted by convolutional layers, although action agreement across image and latent spaces was low.
  • Attention-based Q-learning: The convolutional features were shallow, and the agent did not model interactions between objects.These findings bound the interpretive conclusions of the visual Q-learning approach.
  • Influence analysis: Influence functions identify how removing transitions affects state-action values or policy value, supporting interpretation of off-policy evaluation.The framework includes exact influence algorithms for importance-sampling estimators and fitted Q-evaluation functions.
  • Influence analysis: Influence analysis was used with a cancer simulator and clinicians’ analysis of acute-hypotension management from an ICU dataset.The stated objective was to identify limitations in evaluation and make evaluation more robust.
  • Attention in visual agents: Top-down attention uses recurrent state queries over visual keys and values to produce attention maps showing what and where the agent attends.The model supports sequential, query-dependent selection of task-relevant visual information.
  • Attention in visual agents: Attention maps enabled agents to focus on key features, look ahead along short trajectories, and place tripwires for triggering behaviours.The model was evaluated across a broad range of Atari levels and could select regions and entities according to task demands.
  • Attention in visual agents: Saliency comparisons suggested more comprehensive analysis with attention maps, but their role in reinforcement learning remained mixed when used alone.In combined explanations, saliency maps contextualized important behavioural features.

4.2.8 Conservative Q-Improvement.

Conservative Q-Improvement learns compact decision-tree policies by adding branches when lookahead predicts sufficient reward improvement. Related hierarchical methods represent multi-task policies and learned skills using structured plans and human-language descriptions.

  • Conservative Q-Improvement: Conservative Q-Improvement learns a decision-tree policy for discrete actions and multidimensional state spaces.Its lookahead predicts which split will produce the largest reward increase, starting from a single leaf.
  • Conservative Q-Improvement: CQI grows the tree only when the expected discounted future reward exceeds a dynamic threshold moderated by node visits and split criteria.This yields a direct trade-off between policy accuracy and tree size.
  • Conservative Q-Improvement: In RobotNav, CQI produced substantially smaller trees than the comparison method, with smaller variance and greater reward.The method was evaluated in a two-dimensional navigation environment with obstacle avoidance.
  • Conservative Q-Improvement: CQI did not propose an explanation logic or evaluate the learned policy for interpretability or explainability.Its contribution therefore concerns interpretable policy-model design rather than demonstrated human-facing explanation quality.
  • Hierarchical policies: The hierarchical multi-task framework uses base, instruction, augmented-flat, and switch policies to coordinate previously learned and novel tasks.A stochastic temporal grammar models policy selections and task priorities.
  • Hierarchical policies: In Minecraft experiments with 24 tasks, visualizations primarily demonstrated learning efficiency rather than explanations.The approach was evaluated in a two-room environment and targeted generalization as well as learning efficiency.
  • Hierarchical policies: The framework represents hierarchical plans and skills with natural-language descriptions and can utter human instructions as explanations.Training progressively expands the task set from G0 to G_k and the corresponding policies from π0 to π_k.

4.2.10 LVIN: Imitation learning using the Value Iteration Networks approach.

LVIN decomposes robot planning into a high-level finite-state automaton and a low-level MDP, producing policies expressed through human-interpretable logic states. Its evaluations focus mainly on task performance and model approximation, while interpretability is not directly assessed with human subjects.

  • Model and explanation: LVIN produces desired actions as transitions between human-interpretable logic states, addressing model explanation for discrete actions in 2D spaces.The framework is designed around an interpretable global model rather than only raw neural-network parameters.
  • Model and explanation: LVIN separates planning into an FSA for logical sub-goals and an MDP for robot motion in the physical environment.The FSA represents interpretable propositions and transitions, while the MDP models motion dynamics.
  • Model and explanation: Separate Value Iteration Networks learn transitions and policies for each FSA state from shared expert demonstrations.The high-level FSA and low-level MDP are trained seamlessly using the same demonstrations, making the interpretable model a mimicking process.
  • Evaluation and limitations: LVIN evaluation compares task success with a CNN baseline in lunchbox-packing and cabinet-checking problems, without emphasizing interpretability.The reported evaluation concerns success rates and does not use human-subject assessment of explanations.
  • Evaluation and limitations: The framework requires a hard-coded logic oracle for training, which limits its generality.This limitation concerns the training setup rather than the policy representation itself.
  • Related interpretable models: LMUT offers feature-weight explanations by assigning a linear Q-function model to each leaf and state-partition cell.Its Q-value predictions can be explained through the feature weights of the selected linear model.

4.2.12 Policy level explanations for Reinforcement Learning.

Policy-level explanation methods construct interpretable abstractions of learned policies, including abstract-state graphs, linear programs, and programmatic policies. They summarize decisions through features, transitions, rules, or executable programs, but evaluations often omit direct human assessment of explanation quality.

  • Abstract policy representations: Abstracted Policy Graphs represent policies as Markov chains over abstract states induced by actions in the original MDP.The abstraction groups grounded states treated similarly and predicts transitions between the resulting abstract states.
  • Abstract policy representations: APG Gen records important features for each abstract state, enabling local explanations of which features affect agent decisions.It also summarizes abstract states and their corresponding grounded states.
  • Evaluation: APG evaluations measure feature-importance prediction, multi-step action agreement, and explanation size relative to grounded states.The reported explanation size is sub-linear in the number of grounded MDP states.
  • Methodological pattern: LMUT approximates deep Q-values with interpretable tree-structured linear models, while APG and PIRL provide alternative policy abstractions.These methods are primarily mimicking approaches that learn interpretable models from trained-agent behavior or demonstrations.
  • Programmatic policies: PIRL and NDPS search for high-reward policies represented in domain-specific programming languages, guided by an oracle DRL policy.NDPS uses local search, input augmentation, and policy-distance comparisons to refine programmatic policies.
  • Evaluation: In TORCS, PIRL policies showed smoother trajectories and better generalization abilities than policies discovered by DRL.The review notes that PIRL’s interpretability itself was not evaluated, with or without human subjects.

4.2.14 Understanding Decisions by Interpretable Policy Learning.

Interpretable policy-learning methods translate demonstrated behavior into models that expose beliefs, decision boundaries, logical rules, or flowcharts. They target human understanding of policies, including under partial observability and possible agent bias, with AI-Interpret reporting improved human decision-making over conventional training.

  • INTERPOLE: INTERPOLE jointly models belief updates and belief-action mappings to explain demonstrated decisions through subjective decision dynamics and boundaries.Its explanation logic visualizes belief trajectories and decision boundaries in a low-dimensional belief simplex.
  • Evaluation: Nine clinicians evaluated INTERPOLE, and a majority preferred its explanations of subjective action-belief trajectories and decision boundaries.The evaluation also assessed policy accuracy and the quality of learned belief-update and belief-action models.
  • INTERPOLE: INTERPOLE is designed for partial observability and can model expert policies without assuming unbiased beliefs or optimal behavior.The method operates offline and identifies factors contributing to individual decisions in a language intended for domain experts.
  • Methodological pattern: Both methods are mimicking approaches that derive interpretable models from demonstrated or learned policies rather than directly exposing the original black-box policy.AI-Interpret uses demonstrations from a closed-box RL policy, while INTERPOLE models demonstrated action-observation trajectories.
  • AI-Interpret: AI-Interpret clusters demonstrations, searches for simple logical formulas, converts them into decision trees, and visualizes the resulting strategies as flowcharts.The search is constrained by a domain-specific predicate language, an imitation threshold α, and a maximum rule length d.
  • Evaluation: Across three planning-task types and a human-training experiment, AI-Interpret helped people understand discovered strategies and make better decisions.The review reports that it was more effective than conventional training for improving human decision-making.

4.3 Solving the objectives explanation problem

Objective-explanation methods aim to help people infer an agent’s reward function or intended outcomes, while distal explanations describe long-term dependencies among actions and events. The reviewed approaches incorporate models of human inference, belief maps, causal structures, or opportunity chains, with evidence of improved understanding in selected settings.

  • Objective explanation: Objective explanation methods teach or reveal an agent’s reward parameters by selecting demonstrations that help humans infer its objective function.The reviewed teaching approach explicitly models human beliefs and approximate inference rather than assuming exact inference.
  • Objective explanation: Humans may infer approximately, so objective-teaching methods model approximate inference and select additional trajectories only when they improve reward-parameter inference.The approach evaluates exact and approximate inference models in a driving simulation with ideal users and human subjects.
  • Evaluation: A suitable approximate-inference model significantly improves performance, and coverage with the right model outperforms approaches assuming exact-inference users.The evidence does not imply that every approximate-inference model is better than an exact model.
  • Intended-outcome explanations: Belief maps explain intended future trajectories by assigning values to future state-action pairs and can support tabular and deep Q-learning agents.The method is limited to low-dimensional state-action spaces, and consistency between the belief map and Q is guaranteed only in the tabular case.
  • Distal explanations: Distal explanation models combine decision trees, causal action-influence models, and RNN-learned opportunity chains to express long-term dependencies among actions and outcomes.Opportunity chains describe relations such as one event enabling or causing another, while the models are trained from RL-environment interactions.
  • Evaluation: Across six benchmark domains, distal explanations were robust and accurate, and they performed substantially better than baselines in human-agent collaborative tasks.Human experiments assessed task prediction and subjective explanation satisfaction in Starcraft II tasks.

4.4 Solving the outcome explanation problem

Outcome-explanation methods clarify why agents choose particular actions by exposing reward components, state features, or decision-relevant visual evidence. The reviewed approaches include decomposed-reward explanations, natural-language policy explanations, and visual saliency methods.

  • Reward decomposition: Reward decomposition explains action preferences by comparing semantically meaningful reward types and identifying minimal sufficient positive and negative reasons.Difference explanations compare decomposed Q-vectors for pairs of actions; case studies show these explanations can reveal action-value bugs and optimizer–DRL-loop issues.
  • Reward decomposition: The reward-decomposition approach addresses local outcome explanations by contrasting an action with alternatives in specific states.Its interpretable model represents positive and negative minimal sufficient explanations for applicable action pairs.
  • Natural-language explanations: A natural-language policy-explanation method answers why and how questions about actions under specific conditions using state-based context and communicable predicates.It supports local explanations of policy actions and can represent control logic through Boolean classifiers and template-based responses.
  • Natural-language explanations: The policy-explanation method helps human collaborators shape expectations and debug aberrant behavior without fully understanding the agent’s logic.Applicability was demonstrated across three robotics domains, but experiments emphasized applicability rather than comparative evaluation.
  • Visual explanations: Visual methods explain decisions through object- or action-focused saliency maps, attention masks, and relevance propagation over task-relevant input information.O-DRL links object recognition to saliency visualization, while SARFA focuses maps on the selected action and selective LRP highlights decision-relevant visual input.
  • Visual explanations: Human studies and empirical evaluations report less confusing saliency maps, sharper attention masks, and selective visual explanations across several DRL settings.The reviewed evidence includes chess-puzzle studies, SSINet results, and Atari evaluations of selective LRP.

5 CONCLUDING REMARKS

The review finds substantial progress in explainable deep reinforcement learning, but concludes that the field is still too early for firm conclusions about explainability and transparency. It identifies unresolved evaluation, comprehensiveness, transparency, and development challenges.

  • Scope and status: DRL explainability remains challenging because sequential agents combine closed-box models of environments, objectives, policies, and intertwined decision sequences.These constituent models must support explanations across trajectories rather than isolated predictions.
  • Scope and status: The review concludes that progress exists, but it is too early to draw firm conclusions about DRL explainability and transparency.The authors call for understanding possibilities across the full pipeline and different explainability dimensions.
  • Evaluation gaps: No reviewed work identifies and evidences objective qualities of good RL explanations, and existing evidence from other fields has not been bridged to real-world RL contexts.The authors specifically call for linking explanation effectiveness to deployment characteristics.
  • Evaluation gaps: Many explanation and presentation proposals are fragmentary and ad hoc, with mixed or absent evidence of effectiveness in real-world settings.The review highlights unresolved questions about saliency maps and their combination with contextual methods.
  • Future directions: Future XDRL systems should provide comprehensive, multi-level explanations spanning policies, objectives, local responses, feature importance, and state-action regions.Such explanations should be tailored to deployment contexts, user requirements, constraints, and cognitive aspects.
  • Future directions: The review calls for pragmatic transparency frameworks, principled XDRL development, and further study of mimicking, distillation, and direct interpretable-box designs.Mimicking can add accuracy and fidelity concerns, while many current approaches add explainability layers after advanced DRL development.

A XDRL STATE OF THE ART: METHODS CHARACTERISTICS AND EVALUATION

The survey organizes XDRL methods by explanation problem, paradigm, interpretable model, required DRL access, explanation content, and surface representation, then compares their evaluation and compatibility characteristics. It finds gaps in direct, high-fidelity explanations and in standardized, human-centered evaluation.

  • Survey organization: The review tables classify methods by explanation problem, paradigm, interpretable model, DRL access, explanation content, and surface representation.The evaluation table additionally records goals, explanation-quality assessment, and human involvement.
  • Survey organization: The compatibility table records DRL method, input modality, state and action requirements, and whether explanations are generated online or offline.This profile distinguishes practical deployment assumptions across approaches.
  • Method paradigms: Distillation approaches are underrepresented, motivating more direct use of DRL models for high-fidelity explanations across scales and granularities.The review identifies this as a need for addressing all explainability problems.
  • Method paradigms: Attention models are promising but mainly follow interpretable-box design and have been used mostly for visual agents.This limits their current coverage of policy and outcome explanation settings.
  • Abstraction and evaluation: State, transition, and trajectory abstractions offer generic routes to explanations at multiple granularities and scales.The review recommends extending them to arbitrary hierarchical abstraction levels using learned hierarchical DRL models.
  • Abstraction and evaluation: Explanation evaluations use numerous objectives and measures, making approaches practically incomparable because agreed definitions of explainability, interpretability, and transparency are lacking.Reported human studies also do not involve operators in simulated real-world settings.
Loading 2301.09937v1…