Source-linked AI summary
Explainable Planning
Maria Fox, Derek Long, Daniele Magazzeni
TL;DR
Explainable AI must support trust and human interaction, yet deep neural networks remain difficult to explain and good explanations lack a clear definition. The paper introduces Explainable Planning (XAIP), using planning’s models and decision structures to address explanatory questions. It provides initial results and a roadmap, while noting that a good explanation metric remains undefined.
Problem
Explainable AI needs to support trust and interaction, but deep neural networks remain far from explainable and there is no clear definition of a good explanation.
Method
The paper develops Explainable Planning (XAIP) by using planning models, explanatory questions, and planner behavior to construct explanations.
Results
The paper provides initial XAIP results and a roadmap for effective explanations, including explanations of planner choices and replanning needs.
Takeaways & Limitations
XAIP identifies planning as a contribution to Explainable AI and proposes formalizing user-planner interaction through new constraints and alternatives.
Takeaways & Limitations
There is no clear way to define what constitutes a good explanation, making an explanation metric an important unresolved issue.
Abstract
from arXiv · showhide
As AI is increasingly being adopted into application solutions, the challenge of supporting interaction with humans is becoming more apparent. Partly this is to support integrated working styles, in which humans and intelligent systems cooperate in problem-solving, but also it is a necessary step in the process of building trust as humans migrate greater responsibility to such systems. The challenge is to find effective ways to communicate the foundations of AI-driven behaviour, when the algorithms that drive it are far from transparent to humans. In this paper we consider the opportunities that arise in AI planning, exploiting the model-based representations that form a familiar and common basis for communication with users, while acknowledging the gap between planning algorithms and human problem-solving.
1 Introduction
Explainable AI is motivated by trust, human interaction, and transparency, but producing explanations is harder than making good decisions. The paper presents Explainable Planning (XAIP) as a planning-based contribution with initial results and a roadmap.
- Motivations: Explainable AI is motivated by the need for trust, interaction, and transparency.Trust supports confidence in AI decisions, interaction supports human-AI cooperation, and transparency supports understanding of wrong or differing decisions.
- The explanation challenge: Explainable AI is harder than good decision-making because decisions must be justified in human terms.Even well-understood algorithms such as linear programming can produce results that are difficult to explain.
- The explanation challenge: A central difficulty is determining what an explanation should contain rather than merely translating algorithmic steps into natural language.
- Motivations: Deep Neural Networks remain far from explainable despite major progress and proliferation in machine and deep learning.
- Paper contribution: The paper presents Explainable Planning (XAIP), reports initial results, and proposes a roadmap for more effective and efficient explanations.
- Paper contribution: XAIP applies explainability to AI Planning, including domains where learning is not an option.
2 Related Work
Related work spans plan explanation, plan explicability, natural-language narration, model reconciliation, iterative planning, and explainable agency. This paper goes beyond listing questions by providing initial XAIP results and identifying concrete directions.
- Existing approaches: Plan Explanation translates planner outputs into human-understandable forms and designs interfaces that support this understanding.
- Existing approaches: Plan Explicability models human interpretations to measure and predict the explicability and predictability of plans.
- Existing approaches: Related work also includes natural-language robot-navigation narrations and model reconciliation between differing agent and human models.
- Existing approaches: Planning as an Iterative Process and Explainable Agency address user interaction with planning and agents’ ability to explain decisions.
- Paper contribution: This paper extends discussion of explanation questions by providing initial planning results and pointing to concrete community work for remaining questions.
3 Things to Be Explained
Explainable Planning is organized around questions about planner choices, alternatives, execution deviations, replanning, and uncertainty. Answers should expose relevant knowledge rather than simply narrate algorithmic steps.
- What explanations should provide: An explanation request seeks knowledge the questioner believes the system has but the questioner lacks.
- Question framework: XAIP characterizes explanatory questions and discusses what constitutes a response to them.
- Planner choices: Q1 asks why an action was taken, including which later action it enables or why an alternative would not have been preferable.
- Execution and replanning: Execution questions ask what diverged from expectation and whether an observed divergence makes the plan fail or requires replanning.
- Scope: Explainable Planning must also address questions arising in probabilistic, uncertain, and anytime planning.One example asks whether giving the planner ten more minutes would produce a significantly better plan.
4 Unique Features of Planning
Planning offers several foundations for explanation: explicit models, execution traces, prior plan-explanation work, and transparent decision mechanisms. These features connect planner behavior to domain states, actions, and choices.
- Planning foundations: AI Planning uses techniques that can make complex decision processes easier to understand, even when algorithms and heuristics are sophisticated.
- Planning foundations: Planning models represent domain dynamics and support plan creation, execution, and post-execution analysis.
- Execution traces: Execution traces record observation-action pairs, enabling explanations to focus on state aspects or action choices.
- Related foundations: Mixed-initiative planning has produced a long history of work explaining plans to help humans understand their meaning.
- Decision mechanisms: Transparent planning algorithms make choices deterministic, repeatable, and based on specific mechanisms that can be articulated for users.
5 Providing Explanations
The paper frames explainable planning as answering why actions were chosen, why human alternatives differ, and why replanning or execution changes are warranted. Proposed responses combine causal explanations, human-readable plan representations, replanning after injected decisions, alternative-plan validation, and plan-monitoring techniques.
- Why did you do that?: Explanations must reveal causal links between actions, including dependencies between an early action and a much later plan outcome.The electricity-domain example reverses transformer current early because later peak demand leaves less flexibility to reconfigure flows.
- Why did you do that?: Planning formalisms such as PDDL should be translated into natural-language forms understandable to non-experts.The paper argues that explanations should not simply restate planner algorithms or heuristic terminology.
- Why did you not choose that action?: After a human injects an alternative action, the planner should execute the prefix, apply that action, and replan from the resulting state.Possible outcomes include undoing the human action, returning to the original plan after several steps, finding a different-goal plan, or failing to find a plan.
- Why are planner decisions better?: Alternative planner and human decisions can be compared using metrics different from the search objective, such as evaluating makespan plans by total cost.The proposed integration with VAL evaluates the alternative plan against the new metric after each human-decision injection.
- Why can things not be done?: Explaining why an action cannot be applied requires distinguishing unsatisfied preconditions from actions that prevent goal achievement.VAL can explain the first case, whereas the second requires more challenging justification and may also address unsolvable planning problems.
- Why do I not need to replan?: Execution-time explanations should assess the whole plan when environmental observations differ, so users can understand why replanning is or is not needed.The paper discusses filter-violation monitoring and argues that continuous replanning should be avoided when a plan remains valid despite differences such as longer action durations.
6 Illustrative Examples
The examples show how plan explanations connect actions to goals, justify replanning, and assess human-suggested alternatives. In the AUV domain, a changed connectivity condition identifies why a later action will no longer be executable.
- 6.1 The Rover Domain: The initial explanation that the rock sample enables later communication is improved by considering alternative ways to achieve the supported goal.The example distinguishes a visible consequence from a justification based on the available ways to achieve the communication goal.
- 6.1 The Rover Domain: The rover example uses alternative plans to explain why Rover0 samples the rock at Waypoint0 and why excluding it produces a worse plan.The comparison is based on makespan, while the explanation also notes that Rover1 then performs every action.
- 6.1 The Rover Domain: A follow-up question about Rover1 doing everything can be addressed by requiring the replanned domain to contain an action involving Rover0.The paper describes encoding this requirement with a dummy effect added to Rover0 actions and a corresponding goal.
- 6.1 The Rover Domain: Rover0 is the only rover that can perform the rock-sample mission because the specification prevents it from reaching Waypoint2 and restricts the camera assignment.Rover1 carries Camera1 and is therefore required for the image mission, while Rover0 cannot collect the soil data at Waypoint2.
- 6.2 The AUV Domain: In the AUV example, discovering that waypoints wp32 and wp36 are disconnected triggers a filter violation and explains why replanning is needed.The explanation identifies a later action at time-point 401.494 that will no longer be executable and highlights the changed connectivity condition.
7 Conclusion
The paper concludes that XAIP provides initial results and a roadmap, while identifying unresolved challenges in explanation quality, plan execution, and adapting existing planning work to more complex domains.
- XAIP’s next steps are to formalise the problem and model user–planner interaction through added constraints and explored alternatives.
- Explanations for temporal planning must address choices about the order in which subgoals are achieved.
- Planning under uncertainty raises the problem of explaining observations at execution time that led to a particular planner choice.
- XAIP lacks a clear metric for determining what constitutes a good explanation and should avoid explaining the obvious.
- Existing planning research should be revisited and leveraged because plans are more complex and increasingly used in critical domains.