PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Explainable Artificial Intelligence Improve Human Decision-Making?

Yasmeen Alufaisan, Laura R. Marusich, Jonathan Z. Bakdash, Yan Zhou, Murat Kantarcioglu

arXiv:2006.11194v1cs.LGstat.ML

TL;DR

Prior work has emphasized explainable AI’s interpretability, trust, and usability, leaving its effect on human decision-making uncertain. The paper compares decisions with no AI, AI predictions alone, and predictions with explanations using real datasets. AI predictions tended to improve accuracy, but explanations showed no conclusive added benefit.

  • Problem

    Prior explainable-AI research has focused on interpretability, trust, and usability, with less empirical evidence about effects on human decision-making.

    Source: §1 · p. 2

  • Method

    The authors ran human-subject experiments on real datasets, comparing decisions without AI, with AI predictions, and with AI predictions plus explanations.

    Source: §1 · p. 2

  • Results

    AI predictions tended to improve human decision accuracy, but adding explanations had no significant impact on decisions or ability to outperform the AI.

    Source: §4.5 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    In the tested settings, providing an AI prediction may help human accuracy, while the explanation’s why information did not specifically enhance decision-making.

    Source: §1 · p. 2, Broader Impact · p. 9

    The study used simplified two-choice tasks with lay participants and one explanation method, anchor LIME.

    Source: §2 · p. 2

Abstract

from arXiv · show

Explainable AI provides insight into the "why" for model predictions, offering potential for users to better understand and trust a model, and to recognize and correct AI predictions that are incorrect. Prior research on human and explainable AI interactions has focused on measures such as interpretability, trust, and usability of the explanation. Whether explainable AI can improve actual human decision-making and the ability to identify the problems with the underlying model are open questions. Using real datasets, we compare and evaluate objective human decision accuracy without AI (control), with an AI prediction (no explanation), and AI prediction with explanation. We find providing any kind of AI prediction tends to improve user decision accuracy, but no conclusive evidence that explainable AI has a meaningful impact. Moreover, we observed the strongest predictor for human decision accuracy was AI accuracy and that users were somewhat able to detect when the AI was correct versus incorrect, but this was not significantly affected by including an explanation. Our results indicate that, at least in some situations, the "why" information provided in explainable AI may not enhance user decision-making, and further research may be needed to understand how to integrate explainable AI into real systems.

1 Introduction

Prior explainable-AI research has emphasized interpretability, trust, and usability, leaving its effect on actual human decisions less empirically studied. This paper tests whether explanations improve decision accuracy beyond AI predictions alone or no AI.

  • Prior behavioral research on explainable AI primarily measured understanding, interpretability, trust, and usability rather than decision-making performance.Source: §1 · p. 2
  • The authors argue that interpretable predictions may not improve actual decisions because human choices can reflect heuristics and biases rather than optimality.Source: §1 · p. 2
  • AI predictions tended to help users, but adding explanations did not specifically enhance human decision-making in the two-choice experiments.Source: §1 · p. 2

2 Background and Related Work

The paper situates its experiment amid research on explainable models and human decision-making, where interpretability, trust, and usability have received more attention than objective decision performance. It also notes that explanations and decision aids do not necessarily improve decisions.

2 Background and Related Work

  • The paper characterizes its approach as an experiment with real people and tasks, using simplified decisions and lay participants rather than experts.It uses one explanation method, anchor LIME, rather than comparing explanation types.Source: §2 · p. 2

2.1 Explainable AI/Machine Learning

  • Explainable models include feature-based methods that rank explanatory features and rule-based methods that express predictions as if-then-else rules.Source: §2.1 · p. 2
  • Anchor LIME provides local rules called anchors, while related work also studies additive models and Bayesian rule sets.An anchor sufficiently explains a prediction locally so changes to features outside the rule do not affect that prediction.Source: §2.1 · pp. 2–3

2.2 Human Experiments with Explainable AI and Human Decision-Making

  • Prior human experiments focused largely on interpretability, trust, and usability, while evidence about objective decision-making performance remained limited.Source: §2.2 · p. 3
  • Interpretability alone may not improve decisions, and decision aids have shown mixed effects on human performance.The cited discussion notes that added information does not necessarily lead to better decisions.Source: §2.2 · p. 3

3 Methods

The study uses two real-world datasets and a between-participants experiment to compare human decisions with no AI, AI predictions alone, and predictions accompanied by anchor LIME explanations. Participants completed practice and test trials, making decisions and reporting confidence.

3.1 Dataset

  • The experiments used COMPAS recidivism data and Census income data, selected as understandable real-world contexts that differ in feature count and classifier accuracy.COMPAS contains 6,479 instances and 7 features; Census income contains 32,561 instances and 12 features.Source: §3.1 · p. 4

3.2 Experimental Design

  • A 2 × 3 between-participants design crossed dataset with control, AI-only, and AI-with-explanation conditions.The explanation condition used anchor LIME; the planned sample was 300 participants, or 50 per condition.Source: §3.2 · p. 4
  • Participants made two-choice classifications with 50% chance accuracy, received correctness feedback, and rated confidence on a five-point scale.The AI classifiers were selected separately for each dataset, with overall accuracy of 82% for Census and 68% for COMPAS.Source: §3.2 · pp. 4–5

3.3 Participant Recruitment and Procedure

  • The study recruited 300 Amazon Mechanical Turk participants across six conditions, each completing 10 practice trials and 50 test trials.Median completion time for practice and test trials was 18 minutes.Source: §3.3 · p. 5

4 Results and Discussion

Across the experiment, AI predictions tended to improve decision accuracy, while explanations showed no conclusive added benefit. Accuracy tracked AI accuracy, and participants showed some sensitivity to whether its predictions were correct.

4.1 Participant Decision-Making Accuracy

  • AI prediction improved accuracy over control on Census, but explanation did not significantly outperform AI alone; COMPAS showed no significant differences across conditions.The dataset interaction was significant, and the accuracy increase with AI was driven by Census.Source: Figure 2 · p. 6, §4.1 · p. 6

4.2 AI Accuracy and Participant Decision-Making Accuracy

  • Participant accuracy was strongly associated with mean AI accuracy (β = 0.70, p < 0.001, R2 = 0.28), with positive relationships in both datasets.The AI-accuracy relationship also varied slightly by dataset (β = −0.07, p < 0.01, R2 = 0.03).Source: Figure 3 · p. 6, §4.2 · pp. 6–7

4.2.1 Outperforming the AI Accuracy

  • 7% (14 of 200) of participants had decision accuracy exceeding their mean AI prediction accuracy.The table reports the number of participants exceeding their mean AI accuracy.Source: §4.2.1 · p. 7, Table 1 · p. 7

4.2.2 Adherence to AI Model Predictions

  • Participants matched AI predictions more often when the AI was correct, especially in Census, but explanations did not increase adherence.AI and AI Explanation conditions were collapsed in Figure 4 because their main effect was not significant (p = 0.62).Source: Figure 4 · p. 7, §4.2.2 · p. 7

4.3 Confidence Ratings

  • AI conditions slightly increased confidence versus control, but explanations added no confidence increase; confidence was positively correlated with accuracy (rrm = 0.48, p < 0.001).The correlation was calculated within individuals across the confidence ratings they used.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4 Additional Results

  • Reaction time did not differ by AI condition; Census trials took 1600 ms longer, while self-reported AI use was low despite the accuracy benefit.Only 16 participants said they primarily used, trusted, or followed AI in the AI and AI-explanation conditions.Source: Figure 6 · p. 8, §4.4 · p. 8

4.5 Discussion

  • The findings argue for comparing explainable AI with both no-AI and AI-only conditions, since omitting AI-only could falsely credit explanations for prediction assistance.The authors report that the absence of a significant explanation effect was not due to insufficient power or ceiling performance.Source: §4.5 · p. 8

5 Conclusions and Future Work

The experiments found that AI predictions generally helped decision accuracy, while the advantage of explanations was inconclusive; future work will examine broader goals and settings.

5 Conclusions and Future Work

  • AI predictions alone generally improved human decision accuracy, but the advantage of adding explanations was not conclusive.Participants also tended to follow AI predictions more often when those predictions were accurate.Source: §5 · p. 9
  • Future work will test whether explanations can improve fairness, safety, and ethics by increasing model transparency.Source: §5 · p. 9
  • Planned extensions include studying human–AI error distributions near decision boundaries and interactions in closed feedback loops.Source: §5 · p. 9
  • The authors also plan to expand the datasets to include image data.Source: §5 · p. 9

Broader Impact

The paper examines whether explanations help people improve AI-supported decisions and detect errors, motivated by the limits of relying on predictions alone in consequential settings.

Broader Impact

  • The authors argue that relying on an AI prediction may not be enough for important critical decisions.Source: Broader Impact · p. 9
  • The study finds that AI may improve human decision-making, but explanations may not automatically improve accuracy.Source: Broader Impact · p. 9
  • The authors hope the findings will encourage research on integrating explanations, AI models, and human operators to improve outcomes over either alone.Source: Broader Impact · p. 9

Supplementary Material for "Does Explainable

The supplied material contains a title fragment and arXiv version information.

Supplementary Material for "Does Explainable

1 Age Distribution

The age-distribution figure is accompanied by the observation that most participants were aged 25–34.

1 Age Distribution

2 Self-Reported Decision Strategies

Participants most often reported relying on the data, while primary reliance on AI was uncommon. Accuracy was similar for participants who did and did not report using AI, but lower among those whose strategy reports were blank or nonsensical.

2 Self-Reported Decision Strategies

  • Data-focused strategies predominated: 74 participants reported primarily using data in control, 80 in AI conditions reported data use without mentioning AI, and 16 reported primarily using AI.Partial AI use, including possible use of explanations, was reported by 39 participants.Source: §2 · p. 12
  • Use of AI and data together was more frequent for COMPAS than Census, despite higher mean AI accuracy on Census; strategy counts were generally comparable between datasets.Two raters coded comments independently and resolved disagreements by consensus; inter-rater reliability was κ = 0.70, p < 0.001.Source: §2 · pp. 12–13
  • Mean accuracy was very similar for participants who mentioned using AI versus those who did not; blank or nonsensical strategy reports were associated with lower accuracy, near or at chance (50%).The number of participants with no reported strategy was comparable across the six conditions.Source: §2 · p. 13, Figure 2 · p. 13

3 Speed-Accuracy Tradeoff

Slower participants tended to be more accurate overall, but the speed-accuracy relationship differed by dataset. AI condition had no significant effect on this tradeoff.

3 Speed-Accuracy Tradeoff

  • Each additional second of reaction time predicted a 1.01% increase in accuracy (β = 1.01, p < 0.001, η2 = 0.08).Figure 3 shows the speed-accuracy tradeoff generated by participants in each dataset condition, with shaded 95% confidence intervals.Source: §3 · p. 14, Figure 3 · p. 14
  • The tradeoff varied by dataset (F(1, 288) = 6.65, p < 0.02, η2 = 0.02): Census β = 1.02, COMPAS β = −0.02; AI condition had no significant effect.The relationship was present for Census participants and near zero for COMPAS participants.Source: §3 · p. 14, Figure 3 · p. 14
Loading 2006.11194v1…