Source-linked AI summary
Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery
Devleena Das, Siddhartha Banerjee, Sonia Chernova
TL;DR
Robots can fail during complex tasks, and non-experts may need to understand the cause to help with recovery. This work evaluates scripted explanations and develops an encoder-decoder model to generate them. Context and action history were most effective, and model-generated explanations matched scripted ones in an unseen environment.
Problem
Robots may require non-expert assistance after unexpected execution failures, while existing XAI techniques largely target experts and do not explain such failures.
Source: §1 · p. 1Method
The authors test scripted explanations with different information types, then extend an encoder-decoder model to generate context-based-history explanations from robot states.
Source: §1 · p. 2Results
Takeaways & Limitations
For non-experts assisting with recovery, explanations that combine environmental context and past successful actions are useful, including for internal errors not visible through environmental changes.
Source: §6 · p. 8Context-based-history explanations are not guaranteed to be useful to all non-experts.
Source: §6 · p. 8
Abstract
from arXiv · showhide
With the growing capabilities of intelligent systems, the integration of robots in our everyday life is increasing. However, when interacting in such complex human environments, the occasional failure of robotic systems is inevitable. The field of explainable AI has sought to make complex-decision making systems more interpretable but most existing techniques target domain experts. On the contrary, in many failure cases, robots will require recovery assistance from non-expert users. In this work, we introduce a new type of explanation, that explains the cause of an unexpected failure during an agent's plan execution to non-experts. In order for error explanations to be meaningful, we investigate what types of information within a set of hand-scripted explanations are most helpful to non-experts for failure and solution identification. Additionally, we investigate how such explanations can be autonomously generated, extending an existing encoder-decoder model, and generalized across environments. We investigate such questions in the context of a robot performing a pick-and-place manipulation task in the home environment. Our results show that explanations capturing the context of a failure and history of past actions, are the most effective for failure and solution identification among non-experts. Furthermore, through a second user evaluation, we verify that our model-generated explanations can generalize to an unseen office environment, and are just as effective as the hand-scripted explanations.
1 INTRODUCTION
Robots may fail during complex tasks, and nearby non-experts may need to help recover them by understanding why they stopped. This work introduces error explanations for unexpected failures and studies their content and automated generation.
1 INTRODUCTION
- When robots fail in complex environments, nearby homeowners, medical lab technicians, or warehouse workers may be first to provide assistance.These everyday users are classified as non-experts because they lack formal training in machine learning, AI, or robotics.Source: §1 · p. 1
- Non-experts need to understand why a failure occurred to help with recovery, but existing XAI methods largely target experts and do not explain unexpected failures during plan execution.The paper focuses on failures expected to be within users’ understanding, such as an object being too far away.Source: §1 · p. 1
- The work formalizes error explanations for unexpected failures, evaluates what explanation information helps users identify causes and solutions, and develops an automated generation method.It studies pick-and-place tasks in a home environment and validates the approach through two user studies and computational model analysis.Source: §1 · p. 2
- Explanations combining environmental context with recently successful actions most effectively helped non-experts identify failure causes and solutions; model-generated versions generalized to an unseen environment and matched scripted explanations.The work evaluates scripted explanations, then extends an encoder-decoder model to generate context-based-history explanations.Source: §1 · p. 2
2 RELATED WORKS
Prior work addresses explanations for plans, plan repair, and fault diagnosis, but leaves natural-language explanations of execution failures for non-experts largely unexplored. This paper positions its approach at that intersection.
2 RELATED WORKS
- Prior XAI has emphasized black-box interpretation and classification, while explaining what an action cannot achieve is insufficient when users need to know why it is unattainable.The cited LTL framework explains unattainable actions, and HIGHLIGHTS summarizes agent capabilities.Source: §2 · pp. 2–3
- Earlier XAIP work explains chosen plans or unsolvable plans, whereas this work targets failures that arise during plan execution [12].Related approaches include question-answering, abstraction of unsolvable plans, and justifications for chosen plans.Source: §2 · p. 2
- Plan repair often adapts a failing plan autonomously; human-in-the-loop work has considered missing-item errors or targeted help requests, rather than explaining failures when the robot cannot direct recovery.The present work does not assume the robot knows the correct recovery and can tell the user what to do.Source: §2 · p. 2
- Robotics fault-diagnosis methods support autonomous recovery or expert debugging, while generating natural-language explanations from diagnoses for non-experts remains largely unexplored.Prior methods include unsatisfied-precondition checks, logic inference, Bayesian methods, and neural networks.Source: §2 · p. 2
- Sequence-to-sequence learning previously translated agent states into natural-language explanations in Frogger; this work adapts that approach to continuous robotics [19] [20].Source: §2 · p. 3
3 PROBLEM DEFINITION
The paper extends a planning framework to define explanations for unexpected task failures. It distinguishes plan justifications from explanations that help users understand and recover from a failure state.
3 PROBLEM DEFINITION
- The planning framework represents a problem by a transition function over actions and states, and a planning algorithm maps a problem and desired property to a plan or policy [12].A plan transforms the agent’s initial state into a goal state and has an associated cost.Source: §3 · p. 3
- Eπ justifies a plan satisfying a property, while Eerr is triggered when a failed action causes an unexpected state that halts execution.Eπ applies to a plan as a whole and may be elicited at any time.Source: §3 · p. 3
- Eerr assumes a sound algorithm and plan, with a failure state requiring user assistance; its objective is for users to understand the cause and provide a recovery solution.Examples include an occluded object or a tool out of reach.Source: §3 · p. 3
- The authors first evaluate scripted explanations across information types, then use those results to guide automatic Eerr generation without predefined scripts.The stages are described in Sections 4 and 5.Source: §3 · p. 3
4 INFORMATION TYPES OF E𝑒𝑟𝑟
This section tests combinations of failed action, environmental context, and recent action history as non-expert error explanations. In a simulated Fetch pick-and-place task, context and history improved identification, with context-history explanations especially useful across internal and external errors.
4 INFORMATION TYPES OF E𝑒𝑟𝑟
- The study asks which explanation information helps non-experts identify failure causes and choose recovery solutions, addressing a gap in explanations for unexpected failures during plan execution.The scripted conditions vary the presence of action history and environmental context.Source: §4 · pp. 3–4
4.3 Study Design
- The study crossed action-only versus context-added explanations with absent versus present action history, testing four scripts against no explanation across 30 object-cause failures.Each simulated failure had one cause and one resolution method.Source: §4 · p. 4, §4.3 · p. 5
4.1 Goal, Action Space, & State Space
- The Fetch robot performed a simulated kitchen pick-and-place task, with actions from navigation through grasping and placement and state tracking entities, motion, and action status.The task moved a specified object from the dining table to the kitchen counter.Source: §4.1 · p. 4
4.2 Failure Taxonomy
- Failures are errored plan actions mapped to coarse types, specific causes, and human-selectable resolutions; internal causes are visually hidden, while external causes are apparent in the environment.The fault-tree analysis organizes failures and their repair solutions.Source: §4.2 · p. 5, Figure 2 · p. 5
4.3 Study Design
- Seventy novice participants first established a no-explanation baseline, then judged 12 additional failures with condition-specific explanations or none, receiving accuracy feedback after each response.Participants were recruited through Mechanical Turk; ten participants with perfect baseline accuracy were excluded.Source: §4.3 · p. 5, §4.5 · p. 5
4.4 Measures & Hypotheses
- User benefit was measured as baseline-to-explanation F1 change for failure-cause and solution identification, testing context, history, and internal-versus-external error effects.The analyses compared explanation conditions and causal groups using ANOVA and post-hoc tests.Source: §4.4 · p. 5, §4.5 · p. 5
4.5 User Evaluation of Scripted Explanations
- Context-based explanations significantly improved failure identification over no explanation and action-based explanations, while context also improved solution identification versus none; history improved failure identification and solution identification versus none.The overall history effect on solution identification was not significant, although the history-versus-none comparison was significant.Source: §4.5 · pp. 5–6, Figure 3 · p. 6
- Context-history explanations significantly improved failure identification for both error groups; solution-identification effects were weaker, and internal causes were not visually apparent in the environment.For external failures, CB-H outperformed action-based and no-explanation conditions; for internal failures, it outperformed none.Source: §4.2 · p. 5, Figure 4 · p. 6, §4.5 · p. 6
5 AUTOMATED GENERATION OF E𝑒𝑟𝑟
The authors train an encoder-decoder model to generate CB-H failure explanations from robot state, building on the scripted format found most effective for non-experts. In an office-environment evaluation, participants given generated explanations performed as well as those given scripted explanations.
5 AUTOMATED GENERATION OF E𝑒𝑟𝑟
- The system generates CB-H explanations without templates, focusing on that scripted format because it was most effective in the earlier study.The system can also generate AB, AB-H, and CB explanations.Source: §5 · p. 6, §5.1 · p. 6
5.1 Encoder-Decoder Model Overview
- A GRU encoder embeds environment features; an attention-based GRU decoder combines that context with continuous features and the target object to generate CB-H word by word.Generation ends when the decoder predicts the END token.Source: §5.1 · pp. 6–7
5.2 Feature Set
- The model input combines environment-object embeddings, continuous robot and target-object features, and the desired object; invalid continuous values are masked.Features include distances, robot velocities, action statuses, and whether the target object is in the robot’s area of interest.Source: §5.2 · p. 7
5.3 Data Collection
- The dataset contains 72 kitchen and office simulations, with CB-H failure explanations and natural-language state rationalizations annotated across 2,100 timesteps.There are 60 kitchen and 12 office simulations; rationalizations are used only for model training.Source: §5.3 · p. 7
5.4 Model Training & Evaluation
- Training uses 60 kitchen simulations in grouped leave-one-out cross-validation, with 12 office simulations reserved to evaluate cross-environment generalization.The model selected for the user study was the best-performing model on the office evaluation set.Source: §5.4 · pp. 7–8
- 81.81% accuracy across six failure causes; the model struggled to distinguish two arm-motion causes and sometimes assigned navigation or object-detection failures to unrelated causes.Accuracy required an exact match to the target phrase.Source: Figure 5 · p. 7, §5.4 · p. 8
5.5 User Evaluation of Model Explanations
- Participants given model-generated CB-H explanations performed as well as those given scripted CB-H explanations in failure and solution identification.Generated explanations significantly improved failure identification over no explanations; the difference in solution identification versus no explanations was a trend, while scripted explanations significantly improved it.Source: §5.5 · p. 8, Figure 6 · p. 8
6 CONCLUSION & DISCUSSION
Context and past successful-action history help non-experts identify robot failures and solutions, including internal errors that are not visually observable. The authors also report that model-generated explanations are as effective as scripted explanations and generalize across environments, while noting limits in user coverage, explanation length, and generalization scope.
6 CONCLUSION & DISCUSSION
- Context and past successful-action history most effectively support non-expert failure and solution identification, including for visually unobservable internal errors.The authors report this benefit for context-based-history explanations.Source: §6 · p. 8
- Model-generated explanations are as effective as scripted explanations and generalize across environments.The authors note that broader generalization to additional situations remains to be improved.Source: §6 · p. 8
- Context-history explanations are not guaranteed to help all non-experts, motivating future exploration of user-tailored explanations.The authors suggest tailoring explanations to individual users.Source: §6 · p. 8
- The study assumes explanations can be arbitrarily long, leaving the tradeoff between information and concision for future work.Source: §6 · p. 8
- The encoder-decoder model generalizes across varying failure scenarios, but broader domains, tasks, and failures remain future targets.Source: §6 · p. 8