PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable AI improves task performance in human-AI collaboration

Julian Senoner, Simon Schallmoser, Bernhard Kratzwald, Stefan Feuerriegel, Torbjørn Netland

arXiv:2406.08271v1cs.HC

TL;DR

Black-box AI can prevent domain experts from validating predictions against their knowledge, motivating a test of explainable AI as a decision aid. The paper compares heatmap-supported and black-box AI with domain experts in two preregistered real-world visual-inspection experiments and reports better task performance with explainable AI.

  • Problem

    Black-box AI makes predictions difficult for domain experts to validate, while explainable AI's effect on task performance in real-world human-AI collaboration remains unclear.

    Source: Introduction · p. 3

  • Method

    Two preregistered experiments compare black-box AI with heatmap-based explainable AI among domain experts performing real-world manufacturing and medical visual-inspection tasks.

    Source: Introduction · p. 4

  • Results

    Task performance improved with explainable AI, including reported gains of 7.7 percentage points in manufacturing and 4.7 percentage points in medicine.

    Source: Discussion · p. 13

  • Takeaways & Limitations

    Explainable AI can enable better decisions in human-AI collaboration with benefits for individuals, organizations, and society.

    Source: Discussion · pp. 13, 15

    Both experiments study one visual-inspection work setting and one form of explainability: heatmaps.

    Source: Discussion · p. 14

Abstract

from arXiv · show

Artificial intelligence (AI) provides considerable opportunities to assist human work. However, one crucial challenge of human-AI collaboration is that many AI algorithms operate in a black-box manner where the way how the AI makes predictions remains opaque. This makes it difficult for humans to validate a prediction made by AI against their own domain knowledge. For this reason, we hypothesize that augmenting humans with explainable AI as a decision aid improves task performance in human-AI collaboration. To test this hypothesis, we analyze the effect of augmenting domain experts with explainable AI in the form of visual heatmaps. We then compare participants that were either supported by (a) black-box AI or (b) explainable AI, where the latter supports them to follow AI predictions when the AI is accurate or overrule the AI when the AI predictions are wrong. We conducted two preregistered experiments with representative, real-world visual inspection tasks from manufacturing and medicine. The first experiment was conducted with factory workers from an electronics factory, who performed $N=9,600$ assessments of whether electronic products have defects. The second experiment was conducted with radiologists, who performed $N=5,650$ assessments of chest X-ray images to identify lung lesions. The results of our experiments with domain experts performing real-world tasks show that task performance improves when participants are supported by explainable AI instead of black-box AI. For example, in the manufacturing setting, we find that augmenting participants with explainable AI (as opposed to black-box AI) leads to a five-fold decrease in the median error rate of human decisions, which gives a significant improvement in task performance.

Introduction

Black-box AI makes predictions difficult for domain experts to validate, leaving the effect of explainable AI on real-world task performance unclear. This paper tests whether heatmap explanations help experts use AI predictions more appropriately in manufacturing and medicine.

Contribution

  • Explainable AI is proposed as a decision aid that helps experts follow accurate AI predictions and overrule incorrect ones.Heatmaps make AI predictions more accessible without changing the AI's predictive performance.Source: Introduction · p. 4

Research gap

  • The study addresses limited prior evidence because earlier work often used laypeople, simplified tasks, or designs that did not isolate explainable AI's effect on performance.Research with domain experts and real-world tasks also commonly compared explainable AI with humans alone or omitted real explainable AI.Source: Introduction · p. 5

Novelty

  • The work isolates explainable AI's effect by comparing it directly with black-box AI among actual domain experts performing real-world tasks.This design targets the gap left by studies using nonexperts, artificial tasks, or non-isolating comparisons.Source: Introduction · p. 5

Study design

  • Two preregistered experiments compare randomly assigned black-box and explainable AI treatments in real-world visual inspection tasks.The explainable-AI treatment provides a visual heatmap alongside the AI prediction.Source: Introduction · p. 5
  • Study 1 uses factory workers assessing electronic-product defects, while Study 2 uses radiologists identifying lung lesions in chest X-rays.The manufacturing study includes N = 9,600 assessments, and the medical study includes N = 5,650 assessments.Source: Introduction · p. 5

Expected contribution

  • Participants performed better when supported by explainable AI as a decision aid than when supported by black-box AI.The tasks were designed to represent visual inspection work in manufacturing and medicine.Source: Introduction · pp. 5–6

Results

The results examine how explainable AI affects domain experts’ performance and decision-making across manufacturing and medical visual-inspection experiments. They show improved task performance and changes in radiologists’ precision and adherence to AI predictions without altering disease detection rates or decision speed.

Results

  • Two randomized experiments in manufacturing and medicine required participants to perform visual inspections, comparing black-box AI with explainable AI that provided the same score plus a heatmap.Source: Results · p. 6

Study 1: Manufacturing experiment

  • 96.3% versus 88.6% mean balanced accuracy, with explainable AI also producing a five-fold lower median error rate than black-box AI in manufacturing.The treatment effect was 7.7 percentage points (β = 7.653, SE = 2.178, P = 0.001).Source: Study 1: Manufacturing experiment · pp. 8–9, Figure 2 · p. 10
  • In manufacturing, heatmaps improved adherence to accurate AI predictions and helped workers overrule wrong predictions, identifying 96.9% versus 86.4%.Workers with explainable AI followed accurate predictions at 98.6% versus 93.5%, while black-box users were 3.6 times more likely to erroneously overrule accurate predictions.Source: Study 1: Manufacturing experiment · p. 9
  • Explainable AI improved manufacturing task performance without affecting productivity: median decision speeds were 4.88 s versus 5.01 s, with no significant difference.The speed comparison was not statistically significant (t = 0.308, P = 0.380).Source: Study 1: Manufacturing experiment · p. 10

Study 2: Medical experiment

  • 83.8% versus 79.1% mean balanced accuracy, as radiologists supported by explainable AI outperformed those supported by black-box AI.The treatment effect was 4.7 percentage points (β = 4.693, SE = 1.800, P = 0.01), remaining significant with controls.Source: Study 2: Medical experiment · p. 11
  • 6.4 percentage points: explainable AI significantly improved radiologists’ precision in identifying lung lesions compared with black-box AI (P = 0.014).No difference in disease detection rate was observed; both treatment arms achieved a mean of 90.4%.Source: Study 2: Medical experiment · p. 11
  • 82.1% versus 72.4%: radiologists with explainable AI followed accurate AI predictions more often than radiologists with black-box AI.Radiologists with black-box AI were 54.2% times more likely to erroneously overrule a correct AI prediction.Source: Study 2: Medical experiment · pp. 11–12
  • 50.8% versus 57.7%: radiologists with explainable AI overruled fewer wrong AI predictions than radiologists with black-box AI.These rates indicate that explainable AI supported more appropriate reliance on AI predictions.Source: Study 2: Medical experiment · p. 12
  • 10.71 s versus 10.29 s: decision speed did not differ significantly between black-box and explainable AI treatments (P = 0.348).Thus, explainable AI improved task performance without reducing radiologists’ productivity.Source: Study 2: Medical experiment · p. 12

Discussion

Across two real-world visual inspection settings, explainable AI improved human-AI task performance by helping domain experts use accurate AI predictions and overrule incorrect ones. The findings provide experimental evidence that heatmap explanations can make opaque AI decisions more accessible while retaining human expertise.

Mechanism

  • Heatmaps highlighted areas determining AI predictions without improving the AI’s prediction performance itself.Their contribution was to make existing AI information more accessible for comparison with domain knowledge.Source: Discussion · p. 13

Main findings

  • 7.7 percentage points in manufacturing and 4.7 percentage points in medicine separated explainable-AI from black-box-AI task performance.The results were consistent across factory workers and radiologists performing real-world visual inspection tasks.Source: Discussion · p. 13

Practical outcomes

  • 73.1% of manufacturing participants augmented with explainable AI outperformed the standalone AI algorithm.Factory workers supported by explainable AI identified 13% more defects than peers supported by black-box AI.Source: Discussion · p. 13

Mechanism

  • Explainable-AI users were more likely to follow accurate AI predictions and overrule wrong ones.This behavioral pattern was identified as the prime reason for the performance improvement.Source: Discussion · p. 13

Contribution

  • The study contributes evidence from two preregistered real-world job tasks with domain experts, supporting high external validity and cross-setting consistency.Unlike simplified decision tasks, the experiments involved factory workers in manufacturing and radiologists in medicine.Source: Discussion · pp. 13–14

Limitations

  • The findings are bounded by studying one visual-inspection setting and one heatmap-based explainability approach, motivating replication with other tasks and explanation methods.The authors also caution that explanations can be unreliable or vulnerable to adversarial attacks, while reporting similar results for other heatmap algorithms.Source: Discussion · pp. 14–15

Methods

The methods comprise two preregistered randomized experiments comparing black-box and explainable AI in realistic visual-inspection tasks with manufacturing workers and radiologists. They detail participant criteria, binary performance measures, regression and robustness analyses, and additional assessments of heatmap consistency and benefits for non-experts.

Methods

  • Methods: The researchers preregistered hypotheses and tested them in two randomized experiments approved by ETH Zurich’s Ethics Commission, with informed participant consent.The study design compared explainable AI with black-box AI while complying with ethical regulations.Source: Methods · p. 16

Tasks

Study 1: Manufacturing experiment

  • The experiment used 200 images of four electronic-product types, randomly assigned workers to black-box or explainable-AI treatments, recorded 9,600 product-image assessments, and reported 95.6% balanced accuracy and 92.9% defect detection.Source: Study 1: Manufacturing experiment · pp. 16–18

Study populations

  • Participants had to be adults, and manufacturing participants additionally needed no self-reported visual impairment; preregistered exclusions covered failed tutorials, untimely completion, misbehavior, and extreme performance.Randomization checks confirmed that treatment arms were demographically comparable.Source: Study populations · pp. 19–20

Study 1: Manufacturing experiment

  • The manufacturing experiment used 48 experienced factory workers from a Siemens factory, with exclusions for untimely completion and obvious misbehavior.Participants averaged 13.8 years of working experience, and the available workforce limited the sample size.Source: Study 1: Manufacturing experiment · p. 20

Study 2: Medical experiment

  • The medical experiment recruited 113 U.S. radiologists who assessed chest X-ray images online, after exclusions for noncompletion, lateness, and obvious misbehavior.The study ran from February 27 to March 31, 2024, and radiologists averaged 13.5 years of tenure.Source: Study 2: Medical experiment · p. 20

Statistical analysis

  • Task performance was evaluated against ground-truth labels using balanced accuracy and defect/disease detection rate for binary inspection outcomes.Balanced accuracy equally weights performance across labels to address imbalanced class distributions, while one-sided Welch’s t-tests and OLS models estimated treatment effects.Source: Statistical analysis · p. 21

Robustness checks

  • Robustness checks using precision, participant controls, quasi-binomial regression, and previously excluded participants yielded conclusive findings.Two additional heatmap-generating algorithms also produced similar heatmaps.Source: Robustness checks · pp. 21–22

Comparison to non-experts

  • Non-experts supported by explainable AI achieved 6.3 percentage points higher balanced accuracy than non-experts supported by black-box AI and a higher defect detection rate.The balanced-accuracy improvement was smaller than the 7.7-point increase observed with domain experts.Source: Comparison to non-experts · p. 22

Competing interests

The funding bodies had no control over the research design, conduct, data, analysis, review, reporting, or interpretation.

  • The funding bodies had no control over the research design, conduct, data, analysis, review, reporting, or interpretation.Source: Competing interests · p. 37

Data and code availability

The analyses used specified Python scientific-computing and visualization packages, and the data and code will be publicly released upon publication.

Extended literature review

The extended review distinguishes explainable-AI research focused on computer-science methodological outcomes from this work's focus on behavioral-science outcomes.

A.1 Explainable AI in computer science

This section reviews explainable AI methods and behavioral research on how explanations shape human reliance on AI. It also motivates and describes evaluating these decision aids with domain experts in consequential manufacturing and medical inspection tasks.

A.1 Explainable AI in computer science

  • Explainable AI research distinguishes inherently interpretable models from complex black-box algorithms and develops methods to make AI decision-making more transparent.Examples of interpretable models include linear regression, generalized linear and additive models, decision trees, naïve Bayes, nearest-neighbor, and rule-based methods.Source: §A.1 · p. 40
  • Post-hoc explanation methods help understand black-box models, can be divided into global and local approaches, and categorized as model-specific or model-agnostic, including LIME and SHAP [19].Source: §A.1 · pp. 40–41
  • A plethora of post-hoc explanation methods exists, making method selection difficult and motivating evaluation through fidelity, robustness, human interpretability, and application-grounded human performance.Application-grounded measures assess how humans perform in a certain task.Source: §A.1 · p. 42

A.2 Explainable AI in behavioral science

  • Behavioral-science research has studied human delegation of tasks and decisions to AI algorithms [59, 23, 24, 104], including whether explainable AI increases humans’ likelihood of delegating decisions [25].It also examines algorithm aversion, the phenomenon in which humans are reluctant to use algorithms.Source: §A.2 · p. 43
  • Prior explainable-AI studies often used non-experts and simplified tasks, motivating evaluation with domain experts on real job tasks focused on task performance.Humans must appropriately rely on AI by following correct predictions and overruling wrong ones; domain knowledge may be needed to fully harness explanations.Source: §A.2 · pp. 43–44

B.1 Manufacturing setting

  • Manufacturing inspections are often tedious, tiring, and error-prone, while AI can help factory workers detect defects and focus on value-creating tasks.The study used real-world images of four printed-circuit-board product types, with wrong components, assembly errors, and faulty components represented among the defects.Source: §B.1 · p. 45

B.2 Medical setting

  • Identifying lung lesions in chest X-rays is a difficult but important medical task because subtle lesions can be overlooked and missed detection can delay diagnosis and treatment.The setting motivates providing physicians with a decision aid for chest-X-ray lesion identification.Source: §B.2 · p. 48

C.1 Manufacturing setting

This section examines timed visual-inspection tasks in manufacturing and medicine, comparing interface treatments that provide standalone AI predictions with or without heatmap explanations and recording participants’ assessments and decision speed.

C.1 Manufacturing setting

  • 95.6% balanced accuracy was achieved by the standalone manufacturing AI on 200 out-of-sample product images.The algorithm used a quality-score cutoff of θ∗ = 90 and balanced accuracy accounted for 172 faultless and 28 defective products.Source: §C.1 · p. 51
  • Manufacturing explanations used anomaly heatmaps overlaid on product images, highlighting regions predicted to contain defects.The underlying unsupervised anomaly detector required only faultless training images and could identify previously unobserved defect types.Source: §C.1 · pp. 50–52

C.2 Medical setting

  • 82.2% balanced accuracy and 71.4% disease detection rate were achieved by the standalone medical AI on 50 chest X-ray images.The images were held out from training and threshold selection; Table S3 reports the corresponding confusion matrix.Source: Table S3 · p. 53, §C.2 · p. 53
  • Medical explanations used GradCAM heatmaps overlaid on chest X-rays, with bright regions indicating areas most relevant to predicting lung lesions.Heatmaps were provided only to radiologists in the explainable-AI treatment arm.Source: §C.2 · pp. 53–54

D.1 Manufacturing setting

  • The manufacturing experiment presented 200 visual-inspection images through a computer interface modeled on Siemens’ real-world quality-inspection setup.The procedure included consent, an application tutorial, the inspection task, and a post-experimental questionnaire, with interface versions assigned by treatment.Source: §D.1 · p. 55
  • Participants evaluated 200 product images as faultless or defective within 35 minutes, with final quality assessments and decision speed recorded.The interface varied by treatment, including black-box AI, explainable AI, or no AI.Source: §D.1 · p. 55

D.2 Medical setting

  • Radiologists inspected 50 chest X-ray images within 35 minutes, answering whether each showed at least one lung lesion while recording assessments and decision speed.The explainable-AI interface additionally displayed a heatmap beside the enlarged X-ray view, and treatment assignment determined the interface version.Source: §D.2 · pp. 56–57

E.1 Study 1: Manufacturing experiment

The supplementary analyses compare explainable and black-box AI across manufacturing and medical visual inspection, assess heatmap similarity, and test treatment effects with alternative regression specifications. Results consistently support a significant explainable-AI advantage across reported task metrics, with comparable heatmaps across algorithms.

E.1 Study 1: Manufacturing experiment

  • Manufacturing and medical treatment arms show no statistically significant participant differences in the reported randomization checks.Manufacturing checks cover age, gender, education, and tenure; the medical check covers radiologist tenure.Source: §E.1 · p. 58, §E.2 · p. 58

Supplement F

  • GradCAM heatmaps correlate more strongly with DeepLIFT (r = 0.92) than with LRP (r = 0.63), although all algorithm pairs produce statistically significant correlations.The correlations are based on heatmaps for eight chest X-ray images, and each pairwise correlation is significant at P < 0.001.Source: Robustness of the heatmap · p. 60

metric

  • Precision is reported alongside balanced accuracy and defect or disease detection rate to evaluate participant task performance.The supplementary metric analysis compares explainable-AI and black-box-AI treatments in manufacturing and medicine.Source: metric · p. 62

Supplement H Regression models

  • OLS models may predict balanced accuracies outside the 0–100 range, so the analyses additionally estimate quasi-binomial models with a logit link.Balanced accuracy is bounded between 0 and 100, motivating the alternative specification.Source: Supplement H Regression models · p. 64

H.1 Study 1: Manufacturing experiment

  • All three quasi-binomial manufacturing models find a significant explainable-AI treatment effect for both balanced accuracy and defect detection rate.The models use progressively richer controls, including demographics, tenure, IT skills, and decision speed.Source: §H.1 · pp. 64–65

H.2 Study 2: Medical experiment

The medical regression analyses test explainable-AI treatment effects under multiple control specifications, while exclusion-sensitivity analyses examine whether the findings persist after removing atypical participants. Explainable AI remains significantly associated with balanced accuracy across the main specifications and nearly all exclusion criteria considered.

H.2 Study 2: Medical experiment

  • All three medical OLS models report a significant explainable-AI treatment effect for balanced accuracy.The models add tenure, IT skills, and decision speed as controls across specifications.Source: §H.2 · pp. 65–66

Supplement I Analysis with excluded participants

  • The exclusion analysis removes participants who timed out, showed obvious misbehavior, or exhibited extreme performance according to prespecified criteria.The analysis includes six manufacturing and two medical timeout exclusions, plus additional study-specific exclusions.Source: Supplement I Analysis with excluded… · p. 67

I.1 Study 1: Manufacturing experiment

  • Manufacturing exclusion analyses retain statistically significant explainable-AI effects for balanced accuracy and defect detection rate across combinations of criteria, with one defect-rate exception.The exception occurs when timed-out workers and workers who labeled no product as defective are excluded while extreme performers remain included.Source: §I.1 · p. 67

I.2 Study 2: Medical experiment

The medical exclusion-sensitivity analysis evaluates explainable-AI treatment effects under alternative participant-removal rules. Balanced-accuracy effects are robust to nearly all criteria, whereas disease-detection effects remain nonsignificant.

I.2 Study 2: Medical experiment

  • Medical balanced-accuracy effects remain statistically significant under all exclusion criteria except removing radiologists who assigned one label to every chest X-ray.The corresponding treatment effect is not statistically significant under that single exclusion rule.Source: §I.2 · p. 68
  • The treatment effect for disease detection rate is not statistically significant in the main analysis and does not become significant under alternative exclusion criteria.Thus, the exclusion analysis changes neither the reported nonsignificance nor its direction.Source: §I.2 · p. 68

Experiment with non-experts

A preregistered non-expert experiment compares humans with black-box AI, explainable AI, or no AI in manufacturing inspection. Explainable AI produces higher performance than black-box AI and helps participants follow accurate predictions while overturning incorrect ones, without slowing decisions.

Experiment with non-experts

  • The experiment randomly assigns non-experts to black-box AI, explainable AI, or no AI, yielding 430 final participants and 57,600 product assessments.The AI-treatment arms contain 288 participants.Source: Experiment with non-experts · p. 69
  • Participants supported by explainable AI achieved higher task performance than black-box AI participants across both balanced accuracy and defect detection rate.Black-box AI participants had a mean balanced accuracy of 81.4%, while explainable AI participants had a mean defect detection rate of 77.7% versus 66.4% with black-box AI.Source: Experiment with non-experts · p. 69
  • The explainable-AI treatment effect was statistically significant and large (β = 6.252, SE = 1.733, P < 0.001), improving task performance by 6.3 percentage points.The effect remained significant when relevant control variables, including demographics, self-reported IT skills, and decision speed, were included in the regression model.Source: Experiment with non-experts · pp. 69–70
  • Participants without AI average 72.4% balanced accuracy and are outperformed by both black-box-AI and explainable-AI participants.For defect detection rate, no-AI participants average 53.6% and are also outperformed by both AI treatments.Source: Experiment with non-experts · p. 70
  • Explainable AI increases adherence to accurate quality scores from 92.9% with black-box AI to 95.2%.Black-box-AI participants are also 47.9% more likely to erroneously overrule an accurate AI prediction.Source: Experiment with non-experts · p. 70
  • Explainable-AI participants overrule 79.1% of wrong AI predictions versus 65.8% for black-box-AI participants.The difference is statistically significant (t = 4.563, P < 0.001).Source: Experiment with non-experts · p. 70
  • Decision speed does not differ significantly between black-box AI and explainable AI, with means of 4.61 s and 4.50 s.The comparison is nonsignificant (t = 0.584, P = 0.280), indicating higher performance without a reported speed cost.Source: Experiment with non-experts · p. 71

J.1 Results with precision as task performance metric

Across the supplementary analyses, explainable AI improves precision and decision performance relative to black-box AI, with effects remaining significant across controls and exclusion criteria.

J.1 Results with precision as task performance metric

  • Explainable AI makes non-experts more precise at identifying defective electronic products than black-box AI.Source: §J.1 · p. 71

J.2 Randomization checks

  • Randomization checks find no statistically significant demographic differences among participants in the three treatment arms.Source: §J.2 · p. 72

J.3 Regression models

  • Explainable AI has significant treatment effects on balanced accuracy and defect detection rate across regression specifications with demographic, IT-skill, and decision-speed controls.Source: §J.3 · pp. 73–74

J.4 Analysis with excluded participants

  • The explainable-AI effect remains statistically significant for balanced accuracy and defect detection rate under every preregistered exclusion-criteria combination.Source: §J.4 · p. 74

Preregistered hypotheses

  • Across the manufacturing, medical, and non-expert studies, all preregistered hypotheses were confirmed except non-expert H3 and medical disease-detection-rate H2 and H3.Source: Preregistered hypotheses · p. 76
  • Radiologists supported by explainable AI were significantly more precise at identifying lung lesions than radiologists supported by black-box AI.Source: Preregistered hypotheses · p. 76

Supplement L

Related papers

Loading 2406.08271v1…