Metrics for Explainable AI: Challenges and Prospects

1 Introduction

• Complex and increasingly autonomous machine-learning systems make explainability necessary for decision makers to judge whether AI processes are reasonable and fair.This need is especially acute because such systems are less interpretable, more pervasive, and increasingly entrusted with decisions and actions.Source: §1 · pp. 1–2
• The paper’s central problem is how to measure whether explanations work well and give users a pragmatic understanding of AI systems.It focuses on evaluating XAI systems and human-machine performance through questions about how systems work, what mistakes they make, and why they act as they do.Source: §1 · p. 2

1.1 Key Measurement Concepts

• The XAI process model proposes that instruction and explanatory experience refine users’ mental models, supporting better performance and appropriate trust and reliance.Figure 1 presents the conceptual model and highlights four major classes of measures.Source: §1.1 · p. 3, Figure 1 · p. 3
• The paper hypothesizes that good, satisfying explanations produce better mental models, which in turn support appropriate trust in AI and better performance.It therefore calls for multiple measure types and develops methodological suggestions for evaluating the explanation process, drawing on concepts including causal, abductive, counterfactual, and contrastive reasoning.Source: §1.1 · p. 3

2 Explanation Goodness and Satisfaction

• What counts as an explanation depends on the user’s knowledge, needs, goals, and AI-system context, so explanation seeking can express a need for a particular purpose.Different user questions act as triggers for explanations and goals.Source: §2 · pp. 3–4

2.1 Explanation Goodness

• Explanation goodness concerns clarity and precision judged a priori by independent researchers using a Goodness Checklist, rather than users’ contextualized reactions.The checklist supports designing goodness into generated explanations or evaluating their a priori quality.Source: §2.1 · pp. 4–5

2.2 Explanation Satisfaction

• Explanation satisfaction is the degree to which users feel they understand the explained AI system or process, evaluated after interaction with the XAI system.The scale’s item pool addressed understandability, satisfaction, detail sufficiency, completeness, usefulness, accuracy, and trustworthiness.Source: §2.2 · p. 5, §2.4 · p. 8

2.3 Scale Validation: Content Validity

• Positive CVRs of 0.43–0.60 indicated reasonably high agreement that the Explanation Satisfaction items validly represented the construct.Average CVR was 0.43 for Ph.D. participants, 0.60 for computer scientists, 0.43 for researchers, and 0.57 for others, against a critical value of 0.50.Source: §2.3 · p. 7
• Cronbach’s alpha was 0.86, with item-total correlations of 0.41–0.76 and an average inter-item correlation of 0.71, indicating excellent internal consistency.Item 4 was dropped after participants found its reverse-scored wording confusing; the remaining items were rated essential or useful.Source: §2.3 · pp. 7–8

2.4 Scale Validation: Discriminant Validity

• The Explanation Satisfaction scale was valid overall, distinguishing good from bad explanations with a large effect despite only marginal statistical significance.Good explanations received higher ratings (M = 42.75; SD = 3.40) than bad explanations (M = 30.00; SD = 11.46), with Cohen’s d = 1.5.Source: §2.4 · p. 8

3. Measuring Mental Models

• Mental models are users’ representations of how intelligent systems work, whereas user models are computational models of those mental models used to adapt system interactions.This section focuses on eliciting information about users’ mental models rather than automatically generating user models.Source: §3 · p. 9, §3.1 · p. 9

3.2 Empirical Assertions

• Empirical evidence indicates that guided task reflection can reveal users’ understanding, and different elicitation methods can converge on users’ mental models.People may not report everything accurately without scaffolding, but they can describe relevant knowledge, concepts, and principles when appropriately supported.Source: §3.2 · pp. 9–10

3.3 Overview of Mental Model Elicitation Methods

• Mental-model elicitation includes self-explanation, glitch detection, prediction, ShadowBox comparison, diagramming, and think-aloud tasks, each with distinct analytical demands and limitations.Prediction responses require content analysis and well-justified cases; diagramming can take time, while think-aloud methods provide a quick window but may be incomplete.Source: §3.3 · pp. 10–11

3.4 Application to the XAI Context

• For XAI, preferred methods quickly reveal both strengths and limitations in users’ mental models while producing data that can be readily scored, categorized, or analyzed.Cued retrospection provides scaffolding through post-task probes, prediction tasks can expose counterfactual reasoning, and ShadowBox Lite compares user judgments with expert judgments.Source: §3.4 · pp. 13–14
• Explicit self-explanation improves learning and understanding, but users may overestimate causal understanding; asking them to express reasoning can expose and correct those limitations.ShadowBox Lite offers a rapid self-explanation approach by having users identify what is good and bad in an explanation before comparing their judgments with an expert list.Source: §3.4 · pp. 13–14

3.5 Design Considerations

• XAI evaluations should use multiple mental-model elicitation methods because performance on one task may not align with performance on another or with genuine understanding.A representative group of 10 to 12 participants may suffice for initial elicitation, after which validated performance measures could serve as surrogates.Source: §3.5 · p. 14

3.6 Analysis of User Mental Models

• Mental-model goodness can be analyzed by proposition coding, comparing users’ concepts, relations, and propositions with an expert explanation to estimate correctness and completeness.For example, the cruise-control analysis decomposes explanations into concepts, relations, and propositions, enabling percentage overlap calculations with the expert model.Source: §3.6 · pp. 14–15

3.7 Hypothesis Testing

• The section proposes testing whether performance measures simultaneously indicate the goodness of users’ mental models, rather than assuming that relationship.Comparing high- and low-performing participants’ models with an expert model can provide an empirical test of the hypothesis.Source: §3.7 · pp. 15–16

4.1 Introduction

• Assessing users’ curiosity can indicate whether explanations support insight or instead suppress curiosity and reinforce flawed mental models.Explanations may overwhelm users, hinder question-asking, induce reticence, or increase confusion and complexity.Source: §4.1 · p. 16

4.2 The Nature of Curiosity

• Epistemic curiosity arises from perceived knowledge gaps, novelty, surprisingness, incongruity, or violated expectations, motivating effortful information-seeking to resolve uncertainty.In XAI, recognizing and closing a knowledge gap can make successful explanation or self-explanation seem feasible and produce insight.Source: §4.2 · pp. 16–17
• Existing curiosity instruments are poorly suited to XAI because they measure numerical competence or broad personality traits rather than situation- or task-specific curiosity about computational systems.The proposed method instead asks users to identify the triggers that motivated an explanation request, providing a quick window into task-specific curiosity.Source: §4.3 · p. 18
• Curiosity-checklist responses can constrain explanation generation, identify operations needing explanation, reveal curiosity suppression, and quantify curiosity depth by the number of triggers endorsed.The checklist is administered when participants ask for an explanation.Source: §4.3 · p. 18, Table 7 · p. 18

5.1 Introduction

• Appropriate trust requires users to trust an XAI for specific tasks and contexts while appropriately mistrusting it elsewhere, enabling confident reliance.Measurement must detect mixtures of justified and unjustified trust and mistrust, including breakdowns under time pressure, conspicuous errors, or high false-alarm rates.Source: §5.1 · p. 19

5.2 Explanation as Exploration

• Trusting XAI systems should be explored as context-dependent expectations, helping users verify assertions, anticipate failures, assess uncertainty, and mitigate unjustified trust or mistrust.Users may move from skepticism to justified trust, then to unjustified mistrust after an automation surprise, and eventually toward appropriate trust and reliance through further explanations.Source: §5.2 · pp. 19–20

5.3 Trust Measurement Scales

• Trust-in-automation scales can be reliable, but many are tied to interpersonal, robotic, medical, or experimental contexts and therefore require adaptation for XAI.Minimal measures distinguish trust in outputs from willingness to follow advice; psychometric analyses, including Jian et al. (2000), report reliable scales and high Cronbach alpha values where validity was assessed.Source: §5.3 · pp. 20–21

5.4 Recommendations

• The recommended XAI trust measure combines overlapping items from existing scales and repeats them across trials to track movement toward appropriate trust over time.Measurements can follow individual recommendations or explanations, occur during and after trials, and be integrated to evaluate human–machine performance and trust maintenance.Source: §5.4 · p. 21

6 Measuring Performance

• XAI performance evaluation must consider user performance and human-machine system performance together, including effects of satisfying explanations, mental-model quality, epistemic trust, and exploration of the AI’s competence envelope.Source: §6 · p. 22

6.1 Performance with Regard to the Primary Task Goals

• Primary-task performance can be measured by successful trials within a specified time, tasks completed per period, or trials required to reach a predetermined criterion.Source: §6.1 · p. 22

6.2 Performance with Regard to the User

• User-level performance includes the speed and correctness of predictions about machine outputs, evaluated on typical and atypical cases, plus the correctness and completeness of explanations for rare or anomalous outputs.Source: §6.2 · p. 22

6.3 Performance with Regard to the Work System

• Work-system performance requires trade-off analysis beyond efficiency, including whether explanation satisfaction correlates with mental-model quality or other performance measures and whether trust and reliance are appropriate at competence boundaries.Source: §6.3 · p. 23
• Work-system evaluation can measure controllability and correctability: whether users can produce intended outcomes and readily align machine outputs with objective conditions or their judgments.Source: §6.3 · p. 23
• Productivity can be compared with current practice through learning curves and trials-to-criterion, including conditions where explanation capability is hobbled; adoption and choice behavior also directly assess work-system performance and trust.Source: §6.3 · p. 23

• Measurement methods and ideas remain provisional, with further refinements and extensions expected as XAI projects proceed through complete performance evaluation.The authors present measurement as exploratory and suggest that resolving the metrics challenge may depend on more projects reaching performance evaluation.Source: §7 · p. 24
• The authors distinguish explanation goodness, evaluated a priori by researchers, from explanation satisfaction, evaluated a posteriori by learners.This distinction emerged from their empirical inquiry.Source: §7 · p. 24
• A conceptual model linking explanations, user models, and performance supports a multi-method approach because human-XAI interaction is a complex cognitive system.The authors note that other measurable theoretical concepts may also benefit XAI research and development.Source: §7 · p. 24
• The paper distinguishes a measure from a metric: a measure records performance, whereas a metric applies a context-dependent threshold to guide evaluation or decisions.The same surgery success rate can imply trouble for carpal tunnel surgery below 90 percent but exceptional performance in spinal surgery at 90%.Source: §7 · p. 24
• Operational definitions specify how to measure performance or mental models, but metrics determining whether results are superior, acceptable, or poor come from policy.The authors argue that such metrics do not follow directly or easily from theoretical concepts or operationalized measures.Source: §7 · p. 24

Explanation Goodness Checklist

• The checklist assesses whether an explanation is understandable, satisfying, sufficiently detailed, and complete.It also evaluates whether the explanation is actionable and communicates the system’s accuracy or reliability and trustworthiness.Source: Appendix A · p. 35

Appendix B

• Appendix B presents materials used to evaluate the discriminant validity of the Explanation Satisfaction Scale.Source: Appendix B · p. 36

Explanation Judged a priori to be Relatively "Good"

• The cell-phone directions example asks how cell phones provide directions.Source: Appendix B · p. 36
• The relatively good cell-phone explanation describes GPS positioning, map-service route calculation, transmitted instructions, and real-time step monitoring.Source: Explanation Judged a priori to be… · p. 36

Explanation Judged a priori to be Relatively "Bad"

• The relatively bad cell-phone explanation attributes directions to accurate maps, internet downloads, location awareness, shortest-route calculation, and step-by-step guidance.Source: Explanation Judged a priori to be… · p. 36
• The cruise-control materials contrast relatively good and relatively bad explanations of how the system regulates vehicle speed.Source: Explanation Judged a priori to be… · p. 37

Explanation Judged a priori to be Relatively "Good"

• The relatively good cruise-control explanation describes stored speed, speed sensing, accelerator adjustment uphill, no braking downhill, and brake-triggered disengagement.Source: Explanation Judged a priori to be… · p. 37

Explanation Judged a priori to be Relatively "Bad"

• The relatively bad cruise-control material gives fragmented technical details about pulse-frequency sensing, screw-drive actuation, throttle transmission, engine RPM, and gear adjustment.Source: Explanation Judged a priori to be… · p. 37

Explanation Judged a priori to be Relatively "Good"

• The mathematical account models the atmosphere by dividing the world into small regions described by pressure, temperature, winds, and moisture.It computes conditions at regional boundaries to represent atmospheric changes.Source: Explanation Judged a priori to be… · p. 38
• Strong winds in one region are described as moving air into an adjacent region, illustrating how boundary calculations update the model.The calculations are performed for every boundary between all regions.Source: Explanation Judged a priori to be… · p. 38
• This regional atmospheric simulation is used to predict the path a hurricane will take.Source: Explanation Judged a priori to be… · p. 38

Explanation Judged a priori to be Relatively "Bad"

• The historical account stores previous hurricanes and, after locating a storm from satellite imagery, selects the most frequent path associated with its initial location.The process is repeated hourly as the hurricane moves.Source: Explanation Judged a priori to be… · p. 38
• The historical-path method tracks the hurricane by repeatedly comparing its current location with the database of prior storms.The passage specifies that this tracking process is repeated once every hour.Source: Explanation Judged a priori to be… · p. 38
• Computers also predict when hurricane winds and rain will affect land, triggering hurricane warnings.Source: Explanation Judged a priori to be… · p. 38

Appendix C

• The Explanation Satisfaction Scale asks whether users understand how the tool works and find its explanation satisfying, sufficiently detailed, complete, usable, and useful to their goals.It also asks whether the explanation reveals the tool’s accuracy and helps users judge when to trust or distrust it.Source: Appendix C · p. 39

APPENDIX D

• The trust-scale synopsis defines trust as experience-based positive or negative expectations about system interactions and analyzes it into reliability, predictability, and efficiency.The scale directly assesses confidence, predictability, reliability, safety, and efficiency in an XAI system.Source: Cahour-Forzy (2009) Scale; Adams, et… · p. 41

Cahour-Forzy (2009) Scale; Adams, et al. (2003) Scale

• The Cahour-Forzy and Adams scales are designed for users with considerable prior experience, making them more appropriate after use than immediately after an explanation.Their items were adapted from bipolar agreement ratings to the Likert format used for the Explanation Satisfaction Scale.Source: Cahour-Forzy (2009) Scale; Adams, et… · p. 41
• The trust assessment begins by asking users about their confidence in and feeling of trust toward the tool.This item operationalizes the broader trust construct through a direct user judgment.Source: Cahour-Forzy (2009) Scale; Adams, et… · p. 41

3. Is the [tool] reliable? Do you think it is safe?

• The reliability-and-safety section asks whether the tool is safe and efficient at what it does.These questions extend trust assessment beyond confidence to perceived operational safety and efficiency.Source: §3 · p. 42
• Adams et al. developed separate simulation and generic automation scales, whose items largely overlap with the Cahour-Forzy scale except for liking.The generic scale is intended to evaluate any form of automation.Source: §3 · p. 42
• The automation assessment covers usefulness, reliability, accuracy, understandability, liking, and ease of use.Items use bipolar ratings such as Useful–Not Useful and Reliable–Not reliable on a -5 to +5 scale, followed by ranking the six factors’ importance.Source: §3 · p. 42

• The scale analyzes trust into Fidelity, loyalty, reliability, security, integrity, and familiarity and has been widely used, with later scales adapting its items.These factors were developed through cluster analysis of trust-related words.Source: Jian, et al. Scale (2000) · p. 42
• The integrity item is problematic because machine integrity is not explicated, while familiarity is irrelevant when users’ XAI experience is objectively known.Source: Jian, et al. Scale (2000) · p. 42
• Items 1–4 appear to ask the same thing, indicating substantial redundancy within the scale.Source: Jian, et al. Scale (2000) · p. 42
• Because most items overlap with the Cahour-Fourzy scale, the authors recommend incorporating Jian item 4 into its XAI version.Item 4 is distinctive because it has no counterpart in the Cahour-Fourzy Scale.Source: Jian, et al. Scale (2000) · p. 43

Madsen-Gregor Scale (2000)

• The Madsen-Gregor Scale analyzes intelligent decision-aid trust into reliability, technical competence, understandability, faith, and personal attachment, with alpha = 0.94 and about 70% of variance explained.Its understandability and technical-competence factors may also measure users’ mental models, while faith and personal attachment may better reflect reliance and liking; several items additionally concern usability.Source: Madsen-Gregor Scale (2000) · pp. 43–45

Merritt Scale (2011)

• The Merritt Scale defines trust as reliance on automation under uncertainty and identifies belief, confidence, dependability, propensity to trust, and liking, with Cronbach’s alpha ranging from .87 to .92.It was evaluated in a baggage-screening experiment using a fictitious automated weapon detector.Source: Merritt Scale (2011) · p. 45

Schaefer Scale (2013)

• The Schaefer Scale measures robot trust through ability and performance, but its percentage-based format, anthropomorphic items, and communication items create problems for XAI evaluation.The item “Perform a task better than a novice human user” is identified as potentially useful for the Cahour-Fourzy Scale.Source: Schaefer Scale (2013) · p. 45

Singh, et al. Scale (1993)

• The Singh et al. Scale is inappropriate for users first learning an XAI because it presupposes prior experience and merges trust with reliance in a complacency-oriented measure.Its five factors are confidence, reliance, trust, safety, and complacency.Source: Singh, et al. Scale (1993) · p. 46

Wang, et al. Scale (2009)

• The Wang et al. Scale evaluates trust in a hypothetical combat-identification aid, including deception, mistrust, confidence, dependability, and reliability, but several items are context-specific.The scale’s format is unspecified, and its trust and reliability items overlap with the Cahour-Fourzy Scale.Source: Wang, et al. Scale (2009) · pp. 47–48

Trust Scale Recommended for XAI

• The recommended XAI Trust Scale directly assesses confidence, predictability, reliability, efficiency, believability, wariness, comparative task performance, and liking.Its items draw primarily from the Cahour-Fourzy Scale, with additional adaptations from the Jian, Schaefer, and Madsen-Gregor scales.Source: Trust Scale Recommended for XAI · p. 49
• The recommended scale uses a general Likert format for psychometric consistency and easier rating, but it should be administered after considerable XAI use rather than immediately after an explanation.Its reliability and content validity are inferred from overlap and semantic similarity with existing scales rather than established through direct new validation.Source: Trust Scale Recommended for XAI · p. 49