PaperlayerView exact revision on arXiv

Source-linked AI summary

In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making

Raymond Fok, Daniel S. Weld

arXiv:2305.07722v4cs.AIcs.HC

TL;DR

Studies of AI-advised decision making report inconclusive effects of explanations on appropriate reliance and team performance. This paper proposes a verification-centered account of when explanations may help, concluding that most common task explanations do not support verification and that the distinction between reliance and team performance matters.

  • Problem

    Studies disagree on whether explanations in AI-advised decision making improve appropriate reliance and complementary performance.

    Source: §1 · p. 1

  • Method

    The paper develops a verification-centered theory and distinguishes outcome-graded from strategy-graded reliance.

    Source: §5 · p. 7, §7 · p. 9

  • Results

    Explanations for common tasks often describe the AI’s process rather than enable verification, while extracted answer spans can support complementary performance in factoid question answering from trusted sources.

    Source: §4.1 · p. 5

  • Takeaways & Limitations

    The paper concludes that explanation utility for instance-level decisions depends on whether people can verify the AI’s recommendation, not simply inspect its reasoning process.

    Source: §7 · p. 9

    Strategy-graded reliance does not guarantee complementary performance when humans estimate their accuracy relative to the AI poorly.

    Source: §5 · p. 7

Abstract

from arXiv · show

The current literature on AI-advised decision making -- involving explainable AI systems advising human decision makers -- presents a series of inconclusive and confounding results. To synthesize these findings, we propose a simple theory that elucidates the frequent failure of AI explanations to engender appropriate reliance and complementary decision making performance. We argue explanations are only useful to the extent that they allow a human decision maker to verify the correctness of an AI's prediction, in contrast to other desiderata, e.g., interpretability or spelling out the AI's reasoning process. Prior studies find in many decision making contexts AI explanations do not facilitate such verification. Moreover, most tasks fundamentally do not allow easy verification, regardless of explanation method, limiting the potential benefit of any type of explanation. We also compare the objective of complementary performance with that of appropriate reliance, decomposing the latter into the notions of outcome-graded and strategy-graded reliance.

1 INTRODUCTION

Prior work reports mixed effects of AI explanations in decision making, including failures to improve team performance and increased over-reliance. The paper argues that explanations are useful for complementary performance when they let people verify an AI recommendation, and distinguishes two meanings of appropriate reliance.

1 INTRODUCTION

  • AI-advised decision making seeks appropriate reliance, but studies often find no greater complementary performance than with accuracy or confidence displays, and sometimes find over-reliance.Complementary performance means the human-AI team outperforms either member alone.Source: §1 · p. 1
  • Explanations are useful for complementary performance to the extent that they let decision makers verify whether the AI’s recommendation is correct.The authors contrast verification with explanations that primarily expose the AI’s reasoning process.Source: §1 · p. 1
  • Most studied explanations expose the AI’s reasoning rather than support verification, and human-subject studies rarely find complementary performance except when explanations support answer verification.Explanations of reasoning may still help with debugging or predicting generalization.Source: §1 · p. 1
  • The paper separates appropriate reliance into outcome-graded and strategy-graded reliance to clarify the term’s different meanings.The introduction identifies this distinction as a response to confusion around appropriate reliance.Source: §1 · pp. 1–2

2 BACKGROUND

Prior studies report mixed findings: explanations sometimes accompany complementary performance, but often fail to improve decision making or encourage appropriate reliance. The paper proposes verification as a way to reconcile these results.

2 BACKGROUND

  • Explanations can increase trust even irrespective of their quality, and may introduce confirmation or anchoring biases without necessarily improving performance.The cited studies also report increased perceptions of AI usefulness and decision confidence, which do not necessarily translate into better decisions.Source: §2 · p. 3
  • Proposed explanations for poor results include limited cognitive engagement and concerns about post-hoc explanation fidelity, although the cited interpretability-gap studies found no significant performance impact.The passage presents these as hypotheses and notes that the fidelity concern’s performance impact has not been established in the cited studies.Source: §2 · p. 3
  • Studies disagree on whether explanations produce complementary performance: some report gains, while others find no gain or cannot distinguish explanation effects from AI recommendations plus confidence scores.Some reported gains occurred when the AI significantly outperformed the human, leaving open whether explanations improved collaboration or prompted blind trust.Source: §2 · pp. 2–3
  • The authors frame the mixed findings as a question of when explanations can improve decision making and propose a unifying theory to address it.The theory aims to explain the circumstances, if any, under which explanations improve performance.Source: §2 · pp. 3–4

3 AI-ADVISED DECISION MAKING

The paper focuses on instance-level decisions in which an AI recommends an answer and a human makes the final decision. It evaluates explanations by their effects on team performance and argues that verification, not faithfulness alone, is central to complementarity.

3 AI-ADVISED DECISION MAKING

  • AI-advised decision making is an instance-level process: the AI recommends, then a human makes the final decision using task features and possibly an explanation.The paper limits its argument to decision making, rather than other explanation uses such as model auditing or debugging.Source: §3 · p. 4
  • The analysis focuses on objective efficacy measures such as accuracy, error rate, or decision-making speed, not subjective efficacy, satisfaction, trust, fairness, or understanding.The paper does not attempt to characterize explanation utility for those other objectives.Source: §3 · p. 4
  • Complementary performance means the human-AI team performs better than either the human or AI alone, for example through higher accuracy or cheaper decisions.The paper treats this as a motivating objective for human-AI collaboration, while noting that it is not the only possible measure of usefulness.Source: §3 · p. 4
  • The paper argues that explanation faithfulness is largely irrelevant to decision performance compared with whether the explanation lets the human verify the proposed solution.Explanations may increase trust regardless of correctness, whereas quick verification can let a person recognize a mistake even when the AI is less accurate on average.Source: §3 · p. 4, Figure 3 · p. 4

4 VERIFIABILITY

Explanations are useful for AI-advised decisions to the extent they help people verify recommendations, but verification varies by task and is often unavailable. The section connects this spectrum to human intuitions and distinguishes two definitions of appropriate reliance.

4 VERIFIABILITY

  • Explanations can support complementary performance insofar as they let decision makers determine whether an AI recommendation is correct; feature-importance explanations typically do not enable that verification.They may reveal how features influenced a decision, but often do not show whether the recommendation itself is correct.Source: §4 · p. 4

4.1 Useful Explanations Allow Verification

  • Maze explanations can make a hard-to-find solution quick to check, but this task-specific verification may work only for some answer types.An illegal path does not rule out another valid path, and verification of a candidate solution resembles checking a certificate for an NP-complete problem.Source: Figure 4 · p. 5, §4.1 · p. 5
  • In common XAI tasks, feature-based explanations usually describe the model’s process rather than enable verification; trusted-source factoid answers are a rare exception when highlighted text supports checking correctness.Highlighted sentiment spans may appear reassuring without allowing reliable verification.Source: §4.1 · p. 5

4.2 A Spectrum of Verification

  • Verification lies on a spectrum shaped by the task, explanation, and decision maker; reducing verification cost may suffice, and interface and cognitive effort also matter.The authors conjecture that a single path may not verify which exit has the shortest route, while paths to all exits may support complementary performance.Source: §4.2 · pp. 5–6
  • Explanations may fail to improve performance when plausible errors are hard to spot or when correct answers with flawed explanations are rejected, leaving people to solve the task from scratch.For tasks with irreducible uncertainty, verification’s utility is limited by the stochasticity of outcomes.Source: §4.2 · p. 6

4.3 Verification and Human Intuitions

  • Human intuitions can interact with explanations to reveal model errors or support novel knowledge, yet this approach does not appear to yield complementary performance on most tasks.Verification also presupposes that decision makers have enough task-specific knowledge to characterize solutions.Source: §4.3 · p. 6
  • Outcome-graded reliance judges acceptance against the AI’s post-hoc correctness, whereas strategy-graded reliance defers when the AI is expected to outperform the human.The paper argues outcome-graded reliance is outcome-dependent and nondeterministic.Source: Figure 6 · p. 7

5 WHAT IS APPROPRIATE RELIANCE?

The paper distinguishes outcome-graded reliance, judged by whether people accept correct AI advice and reject incorrect advice, from strategy-graded reliance, judged by expected relative performance. It argues the outcome-graded measure can mislead and recommends strategy-graded reliance or overall team performance as evaluation objectives.

5 WHAT IS APPROPRIATE RELIANCE?

  • Outcome-graded reliance can label indistinguishable cases differently when identical inputs and AI recommendations yield different eventual outcomes.The authors identify this measure as both post-hoc and nondeterministic.Source: §5 · p. 7
  • Strategy-graded reliance accepts AI advice when the AI is expected to outperform the human and rejects it otherwise.The expectation may draw on past performance, instance features, the AI recommendation, or an explanation that supports verification.Source: §5 · p. 7
  • Outcome-graded reliance is a post-hoc measure of behavior, while complementary performance measures the human-AI team’s actual performance.Poor human estimates of relative accuracy may undermine team performance even under strategy-guided reliance.Source: §5 · pp. 7–8
  • 82% for Avery and 55% for Blake: the same policy of following a correct AI and rejecting an incorrect one can yield different team accuracy.Both individuals can have near-perfect outcome-graded reliance despite their different team performance.Source: §5 · p. 8
  • The authors recommend strategy-graded reliance, or avoiding “appropriate reliance,” and identify overall performance as a better objective for evaluating team architectures.Their illustrative case contrasts a 60%-accurate human with a 99.999%-accurate AI when the human is uncertain.Source: §5 · pp. 7–8

6 DISCUSSION

The discussion distinguishes verifiability from faithfulness, considers alternative collaboration designs and other goals for explanations, and relates verifiability to demonstrability in human groups.

6.1 Faithfulness vs. Verifiability

  • For a single isolated decision, verifiability matters more than faithfulness, although faithful explanations may help people model AI capabilities over time.A faithful explanation can be too costly to verify or fail to capture the full decision context.Source: §6.1 · p. 8

6.2 Paradigms of XAI-Advised Decision Making

  • Evaluative XAI presents evidence for multiple options rather than one recommendation, and may help mitigate fixation, anchoring, and automation biases.Cognitive-forcing interventions can increase accuracy at the expense of decision speed, human effort, or satisfaction; interactive explanations can also cost time and effort and reinforce biases.Source: §6.2 · p. 8

6.3 Beyond Complementary Performance

  • Explanations can serve goals beyond complementary performance, including AI-level performance, efficiency, cognitive load, fairness, transparency, debugging, audit, and knowledge discovery.The paper does not characterize explanation utility for these other objectives.Source: §6.3 · pp. 8–9

6.4 Verifiability in Collective Decision Making

  • Human-group research distinguishes intellective tasks with demonstrably correct answers from judgmental tasks without verifiable answers, paralleling the paper’s task distinction.The cited account says a single group member with the correct answer can suffice on demonstrable intellective tasks; recidivism prediction and loan evaluation are examples of judgmental tasks.Source: §6.4 · p. 9

7 CONCLUSION

For instance-level AI-advised decisions, explanations support complementary performance insofar as they let people verify the AI’s recommendation. The paper favors strategy-graded reliance and urges reassessing explanations for verification.

7 CONCLUSION

  • Explanations can engender complementary performance to the extent they let a decision maker verify the AI’s recommendation.The conclusion focuses on instance-level human-AI decision making.Source: §7 · p. 9
  • Most explanations, notably those that elucidate AI reasoning, do not facilitate verification and hence are not helpful for decision making.Source: §7 · p. 9
  • The paper distinguishes outcome-graded from strategy-graded reliance and advocates the latter, which sidesteps nondeterminism and aligns with intuitions of appropriateness.Source: §7 · p. 9
  • The paper urges reevaluating whether explanations integrated into decision-making systems facilitate verification in pursuit of complementary team performance.Source: §7 · p. 10

Related papers

Loading 2305.07722v4…