Source-linked AI summary
The Emerging Landscape of Explainable AI Planning and Decision Making
Tathagata Chakraborti, Sarath Sreedharan, Subbarao Kambhampati
TL;DR
AI systems need to explain their reasoning to people who collaborate with them or rely on their decisions. This survey organizes and compares explainable planning work, then summarizes the field’s emerging trends and the boundaries of its coverage.
Problem
Explaining AI reasoning matters for collaborative problem solving and establishing trust with end users, and the survey focuses on how automated planning addresses this challenge.
Source: §1 · p. 1Method
The survey examines explanation dimensions in automated planning, compares existing approaches, and summarizes emerging XAIP trends.
Source: §1 · pp. 1–2Results
The survey outlines approaches to explaining plans and contrasts them by their properties while summarizing emerging XAIP trends.
Source: §1 · p. 2Takeaways & Limitations
Explanations are relevant to collaboration between people and AI systems and to establishing trust with end users.
Source: §1 · p. 1The survey focuses on explanations of plans as solutions to given planning problems and excludes topics such as failed-plan explanations and execution-time considerations.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In this paper, we provide a comprehensive outline of the different threads of work in Explainable AI Planning (XAIP) that has emerged as a focus area in the last couple of years and contrast that with earlier efforts in the field in terms of techniques, target users, and delivery mechanisms. We hope that the survey will provide guidance to new researchers in automated planning towards the role of explanations in the effective design of human-in-the-loop systems, as well as provide the established researcher with some perspective on the evolution of the exciting world of explainable planning.
1 Introduction
As AI systems increasingly interact with people, explaining their reasoning is important for collaboration and end-user trust, especially in complex planning and decision-making. The survey focuses on automated planning, compares explanation approaches, and outlines the field’s emerging trends and scope.
1 Introduction
- Planning explanations must address complex constraints, mismatches between system decisions and user mental models, and provenance over long-term interactions as the world evolves.The survey contrasts these sequential decision tasks with function-approximation tasks and notes that explanations are often sought for reasoning tasks.Source: §1 · p. 1
- The broader XAI focus has evolved from machine learning toward decision-making, with explainability communities emerging across planning and related fields.The survey situates XAIP among communities and workshops at major AI conferences.Source: §1 · p. 1
- The survey focuses on automated planning, compares existing approaches across explanation dimensions, and concludes by summarizing emerging XAIP trends.It introduces user types and planning-task aspects before surveying work that addresses explanation problems across those dimensions.Source: §1 · pp. 1–2
- The survey covers explanations of plans as solutions to given planning problems, excluding goal reasoning, failed-plan explanations, explicable-plan generation, environment design, and execution-time considerations.It points readers to for a detailed treatment of excluded work on explicable plans and environment design.Source: §1 · p. 2
2 The Many Faces of XAIP
The survey distinguishes explainable-system users by their role in relation to the system: interacting with plans, shaping the model, or developing algorithms.
2 The Many Faces of XAIP
- End users interact with the system and may be affected by a robot’s plans or collaborate with a planner in decision support.Examples include a human teammate in a human-robot team and a user collaborating with an automated planner.Source: §2 · p. 2
- Domain designers acquire the model used by the system, while algorithm designers develop the algorithms themselves.The survey gives goal-oriented conversation-system designers and informed-search developers as examples.Source: §2 · p. 2
- In many real-world applications, domain and algorithm designers are distinct and may have little expertise in common.The survey notes that the distinction is not explicit in and gives as an example of separate expertise.Source: §2 · p. 2
3 The Decision Making Problem
The survey frames planning as producing plans or policies with desired properties, then organizes explanations by process, artifact, and explanatory properties. It connects these choices to the needs of different explainees.
3 The Decision Making Problem
- A planning problem maps available capabilities and states to successor states and costs; plans reach goals, while policies map states to actions.The survey uses this abstraction to convey salient decision-making and XAIP concepts despite task-specific variations.Source: §3 · p. 2
3.1 The Explanation Process
- An explanation responds to an explainee’s question about a solution, often contrasting it with a foil, and continues until the explainee is satisfied.Questions can ask why an action is absent or why a plan was chosen; the foil may be explicit, implicit, or partial [46].Source: §3.1 · p. 2
3.2 Explanation Artifacts: Algorithm/Model/Plan
- Explanations can expose algorithm steps for debugging, model artifacts for end users, or characteristics of the plan itself.The survey notes that model-based explanations are algorithm-agnostic and more useful to end users.Source: §3.2 · p. 2
- The cited approaches distinguish output-level explanations of classification choices (corresponding to plans and also algorithm-dependent learned-model details), purely algorithm-dependent internal representations [50], and algorithm-independent input-data explanations.Source: §3.2 · p. 3
3.3 Properties of Explanations
- Explanations may be social, selective, contrastive, local or global, and abstracted to simplify a model that is too complex for the explainee.Social explanations model user expectations; selective and contrastive explanations choose among or distinguish competing hypotheses [46].Source: §3.3 · p. 3
4 Algorithm-based Explanations
Algorithm-based explanations expose aspects of the planning algorithm, supporting debugging and explanation of decisions from particular algorithms.
4 Algorithm-based Explanations
- Search-tree visualization supports debugging, while other methods explain deep-RL decisions with saliency maps or finite-state policy representations.The finite-state method learns a Moore machine to represent policies learned by recurrent neural networks.Source: §4 · p. 3
5 Model-based Explanations
Model-based XAIP explanations account for gaps between a planner’s reasoning and the user’s mental model or inferential ability. The section surveys dialogue, abstraction, and model-reconciliation approaches, including extensions that handle uncertainty and model-free prediction.
5 Model-based Explanations
- Most XAIP work evaluates solution properties independently of the algorithm, focusing on user inferential ability and mental models; when both align, explanation is unnecessary.These approaches are algorithm-agnostic and model-based.Source: §5 · p. 3
5.1 Inference Reconciliation
- Explanatory dialogue can answer causal-link questions, compare a plan with foils specified as constraints, or derive further plan properties from user-defined properties.Causal-link explanations trace supported preconditions toward the goal; other approaches identify an exemplary plan satisfying foil constraints or use oversubscription planning.Source: §5.1 · p. 3
- For MDP policy questions, explanations describe high-value-state frequency, later desirable actions, or human-understandable reward components.One approach also supplies case-based historical precedents for action outcomes.Source: §5.1 · pp. 3–4
- Unsolvability explanations either propose static initial facts to update as excuses, expose a simpler still-unsolvable abstraction, or provide certificates for designers and debugging.Some abstraction methods use intermediate subgoals to show why foils fail; certificates can be complex and are not intended for end users.Source: §5.1 · p. 4
5.2 Model Reconciliation
- Mental-model gaps can make users reject a planner’s solution; reconciliation is representation-agnostic and has user-study evidence that participants identified the intended decision property.Mental models represent users’ expectations about the planning problem; they may differ from the system model.Source: §5.2 · p. 4
- Model reconciliation treats explanation as updating the system and user models until both support a decision property, with the model difference serving as the explanation.The original approach assumes the mental model is known and searches models induced by the difference between it and the system model.Source: §5.2 · p. 4
- Reconciliation explanations preserve the plan’s property against a foil, while minimal model updates are one selection strategy and users may value equivalent updates differently.Foils may be implicit or explicit, and the framework’s mental model can be represented in different forms, including graphs, planning problems, or logic programs.Source: §5.2 · p. 4
Model Reconciliation Expansion Pack
- Extensions address uncertain or multiple users’ mental models, infer models from foils, assist limited inference, and explore model-free prediction and false explanations.Model-free methods learn how disclosed model information affects whether a user finds a state-action-state tuple explicable; relaxing ground-truth consistency can permit false explanations.Source: Model Reconciliation Expansion Pack · p. 5
6 Plan-based Explanations
Plan-based explanations use plans themselves to communicate observations or make long-horizon solutions more manageable. Approaches verbalize or summarize plans, abstract decision structures, answer policy questions, or invite users to complete partial plans.
6 Plan-based Explanations
- Plan verbalization describes paths along dimensions such as abstraction, specificity, and locality, while domain-independent summarization uses minimal causal links to justify actions.The summarization approach uses model reconciliation with an empty mental model.Source: §6 · p. 5
- Abstraction can simplify policies through low-importance feature projection, extracted temporal subgoals, or semi-aggregated MDPs combining user-specified state aggregation and learned skills.The combined scheme maps policies learned through Deep Q-learning to a semi-aggregated MDP.Source: §6 · pp. 5–6
- Policy questions can use user-specified features or contrast properties true of target traces but not alternatives, while partial plans let users infer completions from task knowledge.The partial-plan approach draws on computational models inspired by inverse reinforcement learning and imitation learning.Source: §6 · p. 6
7 Emerging Landscape
The survey describes a field dominated by explanations produced after planning, alongside work that incorporates the prospect of explanation into decision-making. Recent results suggest this can avoid earlier computational costs and may support more efficient planning in interactive settings.
7 Emerging Landscape
- Although most surveyed work explains decisions after a plan is computed or none is found, recent work folds the possibility of explanation into an agent’s reasoning.The paper notes the potential for more human-aware behavior, drawing an analogy to people making better decisions when asked to explain them.Source: §7 · p. 6
- Recent work shows explanation-aware behavior can be achieved at no greater computational difficulty than classical planning, unlike earlier computationally prohibitive model-space search.Explainable components may also yield more efficient planning in collaborative scenarios or anytime planners preserving high-level constraints in partial plans.Source: §7 · p. 6