PaperlayerView exact revision on arXiv

Source-linked AI summary

Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making

Yunfeng Zhang, Q. Vera Liao, Rachel K. E. Bellamy

arXiv:2001.02114v1cs.AIcs.HC

TL;DR

AI-assisted decision making requires case-specific trust calibration because humans remain responsible for high-stakes decisions and models can err. This paper studies confidence scores and local explanations in two human experiments with comparable human and AI performance. Confidence scores calibrated trust, but calibration alone did not improve joint outcomes, while local explanations showed little trust-calibration effect.

  • Problem

    AI-assisted decisions need case-specific trust calibration so people can distinguish when to follow or distrust model recommendations, but little was known about how confidence scores and local explanations affect trust and actions.

  • Method

    Two human experiments examined confidence scores and local explanations in AI-assisted decision scenarios designed for comparable human and AI performance, including blind delegation.

  • Results

    Confidence scores supported trust calibration, whereas local explanations did not show a comparable trust effect; confidence information produced little improvement in AI-assisted decision outcomes.

  • Takeaways & Limitations

    Trust calibration alone may not improve joint decisions when humans lack enough unique knowledge to complement the AI’s errors.

  • Takeaways & Limitations

    The trust-calibration method depends on model probabilities being well calibrated to true outcome probabilities.

Abstract

from arXiv · show

Today, AI is being increasingly used to help human experts make decisions in high-stakes scenarios. In these scenarios, full automation is often undesirable, not only due to the significance of the outcome, but also because human experts can draw on their domain knowledge complementary to the model's to ensure task success. We refer to these scenarios as AI-assisted decision making, where the individual strengths of the human and the AI come together to optimize the joint decision outcome. A key to their success is to appropriately \textit{calibrate} human trust in the AI on a case-by-case basis; knowing when to trust or distrust the AI allows the human expert to appropriately apply their knowledge, improving decision outcomes in cases where the model is likely to perform poorly. This research conducts a case study of AI-assisted decision making in which humans and AI have comparable performance alone, and explores whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. Specifically, we study the effect of showing confidence score and local explanation for a particular prediction. Through two human experiments, we show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making, which may also depend on whether the human can bring in enough unique knowledge to complement the AI's errors. We also highlight the problems in using local explanation for AI-assisted decision making scenarios and invite the research community to explore new approaches to explainability for calibrating human trust in AI.

1 INTRODUCTION

AI-assisted decision making combines human and model strengths, but success depends on calibrating trust to the model’s case-specific error boundaries. This paper studies whether confidence scores and local explanations improve trust calibration and joint decisions.

  • Human decision-makers need to know when to trust or distrust model recommendations because following likely errors can cause poor or catastrophic outcomes.
  • Trust calibration differs from simply increasing trust: global performance indicators may increase acceptance without distinguishing reliable from unreliable cases.
  • Local explanations are intended to support case-by-case judgments by showing how individual attributes contribute to a prediction.
  • The study examines confidence scores and local explanations as case-specific information that may affect trust and AI-assisted decision outcomes.
  • Confidence information supported trust calibration, but improved calibration did not necessarily improve joint outcomes when humans lacked knowledge that complemented the AI’s errors.
  • Local prediction-specific explanations did not produce a perceivable trust-calibration effect, motivating exploration of alternative explainability approaches.

2 RELATED WORK

Prior work links transparency and accuracy information to trust in AI, but evidence is mixed and may not address prediction-specific trust calibration. This paper distinguishes calibrated trust from generalized confidence in a system.

  • Trusting machine outputs governs behaviors such as accepting recommendations, but trust also depends on disposition and situation awareness.
  • Recent studies found little evidence that greater transparency consistently increases willingness to trust machine-learning systems.
  • Excessive system information may overload users, harm situation awareness, and worsen decision-making outcomes.
  • The paper focuses on calibrating trust—distinguishing situations to trust or distrust an AI—rather than simply enhancing trust in an inferior model.
  • Prior AI-assisted decision studies often found that accuracy indicators and explanations increased trust and sometimes accuracy, but their results may depend on large human–AI performance differences.

3 EXPERIMENT 1: EFFECT OF SHOWING AI CONFIDENCE SCORE

Experiment 1 tests whether confidence scores improve trust calibration and AI-assisted prediction accuracy across recommendation, delegation, and relative-domain-knowledge conditions. The design targets cases where human and AI performance are comparable.

  • H1 predicts that showing AI confidence increases trust more for higher-confidence cases, while H2 predicts improved AI-assisted prediction accuracy.
  • The experiment compares receiving the AI’s prediction with blind delegation, where participants decide whether to delegate without seeing the output.
  • Blind delegation provides a stricter test of trust and trust calibration because delegation occurs without access to the AI’s prediction.
  • The study also examines whether knowing that one has more domain knowledge than the AI affects trust, accuracy, and confidence-based calibration.
  • Participants performed a prediction task designed to produce performance comparable to that of the AI model.

3.1 Experimental Design

The first experiment uses an income-prediction task with controlled confidence, prediction visibility, and model completeness. Participants receive training, domain statistics, and feedback before selecting their own or the model’s final prediction.

  • Participants and task: 72 Mechanical Turk participants completed an income-prediction experiment using demographic and job information.
  • Participants and task: The task predicts whether annual income exceeds $50K using the UCI Adult dataset, whose binary income labels provide ground truth.
  • Participants and task: A profile table presented attribute values with chances out of 10 that matching individuals earned above $50K, supplying single-attribute domain statistics.
  • Experimental factors: The experiment manipulated confidence visibility, AI-prediction visibility, and full versus partial model information in a 2x2x2 design.
  • Experimental factors: Displayed confidence represented the model’s predicted probability for its most likely outcome, expressed as how often the prediction would be correct out of 10 similar individuals.
  • Procedure: Cases were stratified across model confidence levels from 50% to 100% to compare information effects across certainty levels.
  • Procedure: Participants first made their own prediction, then saw the assigned AI information, selected the final prediction, and received correctness feedback across 40 task trials.

3.2 Results

Displaying confidence scores calibrated behavioral trust in the AI across confidence levels, including when predictions were hidden, but this calibration did not improve AI-assisted accuracy in this case study.

  • Trust measures: Behavioral trust was measured through switch percentage and agreement percentage rather than subjective self-reports.Switch percentage counted intentional reliance on the AI when predictions conflicted, whereas agreement percentage included matching predictions automatically.
  • Trust calibration: Showing confidence significantly increased participants’ switching to the AI’s predictions, F(1, 64) = 4.64, p = .035.Showing the prediction and model completeness produced no significant main effects or interactions on switch percentage.
  • Trust calibration: The same confidence-by-confidence-level calibration appeared for agreement percentage, F(4, 256) = 3.82, p = .005, regardless of whether predictions were shown.The confidence-by-prediction interaction was not significant, F(4, 256) = 0.331, p = .857.
  • Accuracy: Participants’ own accuracy averaged 65%, compared with 75% for the AI, so humans and AI had broadly comparable standalone performance in the experiment.The AI advantage was modest and differed from prior work reporting a 37% human disadvantage.
  • Accuracy: Improved trust calibration did not improve AI-assisted accuracy, potentially because human and model uncertainty were correlated and participants lacked sufficiently complementary knowledge.The authors note that lower correlation between human and model uncertainty could permit calibration to improve joint decisions.

4 EXPERIMENT 2: EFFECT OF LOCAL EXPLANATION

Experiment 2 tested whether local explanations calibrate trust and improve AI-assisted decisions. Unlike confidence scores, explanations showed no evidence of improving trust calibration or AI-assisted accuracy.

  • Experiment setup: Local explanations represented each prediction through attribute-level contributions generated with the Shapley method.Orange and blue bars indicated attributes associated with income above or below $50K, respectively.
  • Experiment setup: Experiment 2 used the full-model, with-prediction condition because explanations could reveal the prediction even when it was not explicitly shown.Participants could infer the prediction by comparing the summed orange and blue bars.
  • Trust: Unlike confidence, explanation did not affect participants’ trust in model predictions across confidence levels.Switch percentages remained approximately constant across confidence levels, similar to the control condition.
  • Trust: The confidence condition produced higher switch percentages than both baseline and explanation, with p=.011 and p < .001, respectively.The model-information factor was significant, F(2, 24) = 4.17, p = .028, as was its interaction with model confidence, F(8, 96) = 3.81, p < .001.
  • Accuracy: 63% was average Human accuracy and 75% was AI accuracy in Experiment 2, while AI-assisted accuracy did not differ significantly across information conditions.The test found F(2, 24) = 0.810, p = .457, with a possible reverse trend toward lower accuracy when explanations were shown.
  • Discussion: Local explanations lacked an observed effect on trust calibration and AI-assisted prediction, contrasting with prior findings where indiscriminate trust increases improved outcomes for a much-superior AI.The authors link that contrast to differences in AI-human performance and decision-task setups.

5 DISCUSSIONS

Case-specific confidence information improved behavioral trust calibration, but produced little improvement in joint decision outcomes. The discussion attributes this gap partly to aligned human–AI error boundaries and argues that explainability for trust calibration should target prediction uncertainty and be evaluated with users.

  • Mental Model of Error Boundaries: Case-specific confidence information improved trust calibration, potentially more than model-wide accuracy information.The study measured behavioral manifestations of calibration rather than users’ mental models directly.
  • Alignment of Human’s and AI’s Error Boundaries: Confidence information had little effect on AI-assisted decision outcomes, possibly because human and AI errors were largely aligned.Participants lacked sufficient unique knowledge to complement the AI’s errors in the study setup.
  • Alignment of Human’s and AI’s Error Boundaries: Generalizing AI-assisted decision-making findings requires characterizing human–AI error-boundary alignment, not only overall performance.The authors identify this alignment as an open question for future theories and best practices.
  • Explainability for Trust Calibration: The local explanations tested did not produce a perceivable trust-calibration effect because sensible top features appeared regardless of prediction confidence.Inferring confidence by summing positive and negative feature contributions may not have been obvious to people without machine-learning training.
  • Explainability for Trust Calibration: The authors call for explainability methods sensitive to model performance or prediction uncertainty and evaluated with targeted users in the loop.These goals extend beyond existing metrics such as faithfulness, human understanding, or acceptance.

6 LIMITATIONS

The study’s limitations concern participant expertise, the contrived task setting, and dependence on well-calibrated model probabilities.

  • Participant Expertise: Participants were not experts in income prediction, although training and domain statistics brought their accuracy to within 10% of the model’s.The authors describe this as a human-grounded evaluation using lay participants as behavioral proxies.
  • Task Setting: The prediction task was contrived, and participants were not held responsible for outcomes.An outcome-based bonus was used to motivate decision optimization, while more consequential studies would require ethical caution.
  • Confidence Calibration: Showing prediction confidence depends on the model’s predicted probabilities being well calibrated to true outcome probabilities.The authors note that methods such as Platt Scaling or Isotonic Regression may address this issue for some models.
Loading 2001.02114v1…