PaperlayerView exact revision on arXiv

Source-linked AI summary

How model accuracy and explanation fidelity influence user trust

Andrea Papenmeier, Gwenn Englebienne, Christin Seifert

arXiv:1907.12652v1cs.CYcs.AIcs.LG

TL;DR

Complex machine-learning systems need explanations, but persuasive explanations may not faithfully represent their models. This paper tests accuracy and explanation fidelity in a nine-condition user study and finds that accuracy matters more for trust, explanations can reduce trust, and users do not trust bad classifiers merely because explanations are provided.

  • Problem

    It is unclear how explanation fidelity affects user trust when explanations may appear plausible without accurately representing complex machine-learning models.

    Source: §1 · p. 1

  • Method

    The authors validate high- and low-fidelity explanations and compare three classifier accuracies with three explanation types in a user study measuring observed and self-reported trust.

    Source: §3 · p. 3, Measures & Analysis · p. 3

  • Results

    Accuracy influences trust more strongly than explanation fidelity; explanations can harm trust, and high-fidelity explanations do not make users trust a bad classifier.

    Source: §1 · p. 1, §5.2 · p. 6, §6 · p. 6

  • Takeaways & Limitations

    Minimum explanations may harm rather than improve trust, so explainability should be evaluated together with model accuracy and both behavioral and self-reported trust.

    Source: §1 · p. 1, §6 · p. 6

    The findings cannot be generalized across cultural backgrounds because participants were mainly Caucasian.

    Source: §5.4 · p. 6

Abstract

from arXiv · show

Machine learning systems have become popular in fields such as marketing, financing, or data mining. While they are highly accurate, complex machine learning systems pose challenges for engineers and users. Their inherent complexity makes it impossible to easily judge their fairness and the correctness of statistically learned relations between variables and classes. Explainable AI aims to solve this challenge by modelling explanations alongside with the classifiers, potentially improving user trust and acceptance. However, users should not be fooled by persuasive, yet untruthful explanations. We therefore conduct a user study in which we investigate the effects of model accuracy and explanation fidelity, i.e. how truthfully the explanation represents the underlying model, on user trust. Our findings show that accuracy is more important for user trust than explainability. Adding an explanation for a classification result can potentially harm trust, e.g. when adding nonsensical explanations. We also found that users cannot be tricked by high-fidelity explanations into having trust for a bad classifier. Furthermore, we found a mismatch between observed (implicit) and self-reported (explicit) trust.

1 Introduction

This paper studies how classifier accuracy and explanation fidelity affect user trust in automatic decision systems. It combines validated explanations with a user study to examine these effects and their interaction.

  • The study investigates how explanation fidelity influences trust in automatic decision systems, addressing whether plausible explanations truthfully represent the underlying model.Source: §1 · p. 1
  • A user study compares classifier–explanation combinations in a social-media task, using three accuracy levels and validated high- and low-fidelity explanations.The study involves 327 participants and measures trust and perceived understanding.Source: §1 · p. 1
  • Accuracy is the strongest determinant of trust: higher-accuracy systems receive higher trust, while explanation effects depend on model accuracy and user consciousness.Source: §1 · p. 1
  • For medium-accuracy systems, high-fidelity explanations do not harm trust but low-fidelity explanations do; for high-accuracy systems, either explanation type decreases trust.Source: §1 · p. 1
  • The study reports a discrepancy between users’ observed behavior and self-reported trust, motivating objective trust measures alongside questionnaires.Source: §1 · p. 1

2 Related Work

Prior work frames explanations as a way to improve understanding and trust, but emphasizes that explanations may be persuasive without faithfully describing a model. The paper therefore motivates evaluating trust through both subjective reports and observed behavior.

2 Related Work

  • Explainable AI addresses the need to communicate how complex systems reach decisions, especially where fairness, correctness, safety, or sensitive data are involved.Source: §2 · p. 2, §2.1 · p. 2

2.1 Explanations in AI

  • Minimum explanations connect input features to predictions, while model-agnostic and salience-based methods identify decisive features for human inspection.Source: §2.1 · p. 2
  • Explanation fidelity is not guaranteed by plausibility: retrospectively added explanations can appear convincing while failing to represent the underlying model.Source: §2.1 · p. 2

2.2 Trust in AI

  • Trust is treated as a subjective assurance that a system performs as expected, and misclassifications can reduce it.Source: §2.2 · p. 2
  • Because explanations may affect behavior differently in mindful and mindless states, trust should be evaluated with both questionnaires and observation.Source: §2.2 · pp. 2–3

3 Study Design

The study uses a between-subjects online experiment with nine classifier–explanation conditions and repeated classification judgments. Trust is assessed through both self-report and changes in users’ classifications.

3 Study Design

  • The design crosses three classifier accuracy levels with high-fidelity, low-fidelity, and absent explanations, producing nine experimental conditions.Participants are randomly assigned to one condition and one of ten disjoint Tweet subsets.Source: §3 · p. 3, Table 1 · p. 3

Apparatus & Procedure

  • Participants first classify 15 Tweets manually, then classify the same Tweets with an automatic system before completing trust and understanding measures.Source: Apparatus & Procedure · p. 3

Measures & Analysis

  • Observed trust is inferred from changing classifications toward the system’s prediction and away from the truth, whereas opposite changes indicate mistrust.Source: Measures & Analysis · p. 3
  • Self-reported trust averages 19 questionnaire items, while perceived understanding averages three five-point Likert ratings.Source: Measures & Analysis · p. 3

Participants

  • 327 participants completed the study, with 286 valid cases remaining after attention-check and completion exclusions.Source: Participants · p. 3

4 Experiment

The experiment builds three accuracy-controlled classifiers, generates high- and low-fidelity feature explanations, samples balanced Tweet subsets, and validates explanation fidelity before the user study.

Dataset

  • The dataset contains 4,324 Tweets with 1:1 class balance, using only items with unanimous agreement among at least three annotators.Source: Dataset · p. 3

Classifiers

  • The classifiers achieve test accuracies of 0.97, 0.76, and 0.03, with the low-accuracy model created by training the high-accuracy setup on inversed labels.Source: Classifiers · pp. 3–4
  • High-fidelity explanations select decisive words using model-based methods, whereas low-fidelity explanations randomly select visually similar words and may provide no highlighted words.Source: Classifiers · p. 4

Subset Sampling

  • The study samples 15-Tweet subsets that are non-overlapping, class-balanced, accuracy-matched to the test set, and selected for close training-set feature distributions.Source: Subset Sampling · p. 4

Explanation Evaluation

Graphical User Interface

  • The experiment presents decisions and explanations in a minimalistic web interface designed for social-media administrators detecting offensive content.Source: Graphical User Interface · p. 4

5 Results and Discussion

Across the study, model accuracy influenced trust more strongly than explanation fidelity, while explanation effects depended on accuracy and users’ level of consciousness. Self-reported and observed trust did not always align, and the study’s conclusions are bounded by its measures and participant sample.

5.1 Model Accuracy

  • Higher accuracy consistently corresponded to higher self-reported trust: no C0.03 system outranked C0.76 or C0.97, and no C0.76 system outranked C0.97.Users also rated low-accuracy systems as less predictable, although C0.03 and C0.97 were objectively equally predictable.Source: §5.1 · p. 5, Figure 2 · p. 5, Table 3 · p. 5

5.2 Explanations

  • Explanations did not generally increase self-reported trust: they decreased trust for C0.97, had no effect for C0.03, and low-fidelity explanations reduced trust for C0.76.For C0.76, high-fidelity explanations did not significantly reduce trust, whereas low-fidelity explanations produced a lower score.Source: §5.2 · pp. 5–6, Figure 2 · p. 5
  • 0.76 had the highest trust rate, while Clow 0.76 had the highest rate of mistrust; C0.03 showed no diversion, observed trust scores were homogeneous, and users did not trust a bad classifier regardless of explanation.Source: §5.2 · p. 6

5.3 Objective Trust Measure

  • Observed behavior diverged from self-reports: users gave C_no0.97 the highest self-reported trust but followed C_high0.97 more often than C_low0.97.The authors note that observed trust was ambiguous and highly variable, requiring validation in future research.Source: Figure 3 · p. 5, §5.3 · p. 6

5.4 Limitations

  • The study used minimum input-output explanations, and its results cannot be generalized across cultural backgrounds because participants were mainly Caucasian.The influence of task difficulty and explanation visualization remains for future investigation.Source: §5.4 · p. 6

6 Conclusion

This study provides empirical evidence that model accuracy and explanation fidelity shape user trust, with accuracy having the strongest impact. Explanations can harm trust, fidelity matters especially for medium-accuracy systems, and self-reported trust can diverge from observed behavior.

  • Model accuracy had the strongest overall impact on user trust.The study compared three accuracy levels and three explanation-fidelity levels across nine conditions.Source: §6 · p. 6
  • Participants trusted systems without explanations most, showing that minimal explanations can harm rather than improve trust.The authors attribute this to reconciling conflicting information between users’ mental models and explanations.Source: §6 · p. 6
  • For medium-accuracy systems, high-fidelity explanations did not harm trust, whereas low-fidelity explanations decreased it.The reported medium-accuracy condition was classifier C0.76.Source: §6 · p. 6
  • Users’ awareness level influenced perceived trust, and self-reported trust differed from trust inferred from participants’ actions.The authors call for further research using richer explanations, investigating trust factors, and studying trust over time.Source: §6 · p. 6
Loading 1907.12652v1…