PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making

Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, Xiaojuan Ma

arXiv:2301.05809v1cs.HCcs.AIcs.LG

TL;DR

AI-assisted decisions require people to know when to trust AI and when to trust themselves, but prior calibration relied on AI correctness likelihood while overlooking human correctness likelihood. This paper models human correctness likelihood from approximated decision-making rules and similar cases, then uses three strategies to calibrate trust; a 293-participant experiment found more appropriate AI trust than AI-confidence-based methods, with improved team performance.

  • Problem

    Prior trust-calibration methods use AI confidence as AI correctness likelihood but overlook human correctness likelihood, although human self-confidence may be poorly calibrated.

  • Method

    The paper approximates individual decision-making models, estimates human correctness likelihood from similar instances, and proposes three strategies for exploiting human-AI likelihoods.

  • Results

    The three strategies produced more appropriate human trust in AI than baseline conditions, especially when AI recommendations were wrong, and improved team performance.

  • Takeaways & Limitations

    Calibrating trust from both human and AI capabilities offers a framework for more human-compatible AI-assisted decision-making.

  • Takeaways & Limitations

    The experiments approximate human decision-making models with decision rules, which provide only a general model and cannot cover all cases.

Abstract

from arXiv · show

In AI-assisted decision-making, it is critical for human decision-makers to know when to trust AI and when to trust themselves. However, prior studies calibrated human trust only based on AI confidence indicating AI's correctness likelihood (CL) but ignored humans' CL, hindering optimal team decision-making. To mitigate this gap, we proposed to promote humans' appropriate trust based on the CL of both sides at a task-instance level. We first modeled humans' CL by approximating their decision-making models and computing their potential performance in similar instances. We demonstrated the feasibility and effectiveness of our model via two preliminary studies. Then, we proposed three CL exploitation strategies to calibrate users' trust explicitly/implicitly in the AI-assisted decision-making process. Results from a between-subjects experiment (N=293) showed that our CL exploitation strategies promoted more appropriate human trust in AI, compared with only using AI confidence. We further provided practical implications for more human-compatible AI-assisted decision-making.

1 INTRODUCTION

The paper addresses the limits of calibrating trust solely from AI correctness likelihood by estimating both human and AI capability at each task instance. It introduces a framework and three exploitation strategies that improve appropriate trust and team performance relative to AI-confidence-based methods.

  • Motivation: Prior approaches calibrate human trust mainly from AI confidence, often using a preset threshold to encourage trust or distrust.Well-calibrated AI confidence can represent AI correctness likelihood, but empirical findings from these approaches are mixed.
  • Motivation: Humans’ correctness likelihood is often overlooked, despite people’s poorly calibrated self-confidence and the need to decide case by case whether to rely on AI.The paper frames this as a comparison between the relative capabilities of humans and AI on each task instance.
  • Framework: The proposed framework calibrates trust using predicted human and AI correctness likelihood at the task-instance level rather than AI confidence alone.The framework aims to promote appropriate human trust and complementary team performance.
  • Human capability modeling: Human correctness likelihood is estimated by approximating an individual’s decision-making model and applying it to similar task instances to calculate probable performance.The method combines data-driven initialization with interactive modification of possible decision rules.
  • Trust calibration strategies: The paper proposes three strategies—Direct Display, Adaptive Workflow, and Adaptive Recommendation—to communicate human-AI correctness likelihood explicitly or implicitly.These strategies are designed to exploit both parties’ capability estimates during AI-assisted decision-making.
  • Evaluation: In a between-subjects crowdsourcing experiment with 293 participants, the strategies produced more appropriate AI trust, especially for wrong AI recommendations, and improved team performance.Most subjective perceptions or experiences did not differ significantly across conditions.

2 RELATED WORK

Related work commonly calibrates trust by conveying AI capability, especially through calibrated confidence, explanations, or interface interventions. The paper identifies two gaps: unreliable human self-confidence and trust calibration that ignores human correctness likelihood.

  • Trust calibration: Trust calibration concerns correspondence between people’s trust in AI and the AI’s actual capabilities.Under-trust can lead to disuse, while over-trust can lead to misuse.
  • AI capability cues: Prior work conveys AI capability through accuracy, explanations, observed outputs, or calibrated confidence, with confidence representing AI correctness likelihood in specific instances.Several studies use AI confidence to shape trust or decision-making support.
  • Prior interventions: Existing confidence-based interventions include directly displaying confidence, allocating more decision time at lower confidence, and showing alternative predictions below a threshold.These designs target trust calibration, anchoring, or reliance on AI suggestions.
  • Open limitations: The paper identifies a limitation in assuming that humans can accurately perceive their own capability after observing AI confidence.Subjective self-confidence usually cannot accurately represent actual human correctness likelihood.
  • Open limitations: A second limitation is that prior approaches calibrate trust from AI correctness likelihood while ignoring human correctness likelihood in each case.The paper asks whether people should doubt AI when AI’s likelihood is low but the human’s is even lower.
  • Human mental models: To address these gaps, the paper combines data-driven initialization with interactive rule modification to derive individual decision-making mental models.The approach is intended to save users’ time while retaining opportunities for manual refinement.
  • Adaptive interaction: The paper incorporates cognitive biases into adaptive interaction design while regulating trust according to the correctness likelihood of both parties.When AI correctness likelihood is higher, the design uses cognitive processes to support greater AI reliance.

3.1 Overall process of human correctness likelihood modeling

The human correctness-likelihood modeling process estimates a person’s capability for a new case from decisions on similar instances. It approximates individual decision rules, applies them to neighboring cases, and computes a distance-weighted estimate.

  • Likelihood estimation: For a new case, the method applies the approximated model to neighboring cases, computes possible performance against ground truth, and estimates current-case correctness likelihood with distance weighting.Ground truth is available for the task dataset but not for the current case.
  • Overall process: The method estimates human correctness likelihood from past performance on similar task instances.This is motivated by the view that people weigh similar past experiences when making decisions.
  • Overall process: Because sufficient human decision data may be unavailable, the method first approximates each person’s decision-making model as a mapping from task input to human decision.The model is then used to predict possible decisions in similar instances.
  • Model construction: The modeling procedure combines data-driven initialization with interactive modification to derive decision rules employed by each individual.The paper presents this procedure as a four-step process.
  • Model construction: Users first provide predictions on a small sample, after which a classic decision tree infers initial decision models and rules.The initial model is based on limited user decision data.
  • Model refinement: Users interactively revise the initial model so that its rules better align with their inner decision-making process.This step compensates for the possibility that limited training data produces an inaccurate initial model.
  • Validation: Two small-scale preliminary studies examine the first three steps and the fourth step separately.The studies are labeled Study I.1 and Study I.2.

3.2 Task setup

The study used income prediction from the Adult Income dataset as a low-risk, low-expertise testbed, with calibrated AI confidence and human decision data collected to estimate human correctness likelihood.

  • Task and dataset: Income prediction asked participants to classify whether annual income exceeded $50K using demographic and job information.The task used the Adult Income dataset from the UCI Machine Learning Repository.
  • Task and dataset: The task was selected because it required little domain expertise and imposed relatively limited risks.
  • Task and dataset: The study presented five features—age, education, occupation, marital status, and weekly work hours—to non-expert participants.These were selected from 14 attributes using feature permutation importance.
  • Task cases: The 40 task samples were split into two batches: the first elicited human decisions for model construction, and the second supported the main decision task.Presentation order within each batch was randomized, while the batches were fixed across participants.
  • Task cases: AI confidence was controlled to match actual correctness likelihood, with low-confidence samples averaging 0.6 and high-confidence samples averaging 0.8.In low-confidence samples, 6 of 10 predictions were correct, corresponding to 60% CL.
  • Human CL motivation: Participants’ self-reported confidence was not a reliable human CL indicator because it was poorly calibrated and unrelated to actual accuracy.The authors therefore modeled human CL from decision-making behavior rather than self-confidence.

3.3 Study I.1: Comparison of interfaces for users to specify their decision-making models

The authors compared interactive decision-tree and rule-set interfaces for helping users specify their decision-making models. The rule-set interface better matched reported human reasoning and was adopted for the final study.

  • Interface comparison: The study compared an interactive decision tree with an interactive rule set for representing and updating human decision-making models.Rules were considered suitable for approximating human decision-making, while humans often use tree-like structures.
  • Interface design: Both interfaces initialized a decision tree from participants’ first 20 human-only decisions and allowed users to revise the model.The tree interface exposed nodes, whereas the rule-set interface presented converted if-then rules.
  • Study findings: Seven of 10 decision-tree users reported that a single tree could not represent their actual decision process.One participant described switching between age and education as the primary criterion.
  • Study findings: Three of 10 decision-tree users found the tree visually complex, including difficulty reading it quickly.
  • Study findings: Users regarded the interactive rule set as more visually interpretable and more consistent with their decision-making process.The authors consequently used it in the final version for interactive model revision.
  • Limitations: Rule-based modeling remained limited by uncovered edge cases, intuitive decisions, and the difficulty of forming accurate rules from few samples.The system used the initialized model for some edge cases.

3.4 Study I.2: Performance testing of our human correctness likelihood estimation method

The authors estimated human CL by applying revised decision models to similar cases and weighting their predicted correctness by similarity. The estimate correlated with human accuracy and better identified which teammate was more likely to be correct than AI confidence alone.

  • CL estimation: Human CL was estimated from a user-revised decision model’s possible predictions on N similar task instances.The method compares those predictions with known ground-truth labels.
  • CL estimation: The number of neighbors may need to differ across tasks because too many introduce dissimilar samples and too few produce unstable accuracy estimates.
  • CL estimation: More similar neighbor instances received greater weight, while distant instances contributed progressively closer to random-guessing performance.The study set N to 10 as a trade-off between similarity and coverage.
  • Validation: Human-revised models achieved 80.7% prediction accuracy on the last 20 task instances, versus 77.5% for auto-generated models.The improvement was slight and not significant, while paired tests showed the CL method was more effective than AI confidence alone on complementary cases.
  • Validation: 76.4% of complementary cases were recalled by the human-AI CL method, compared with 66.7% by the AI confidence-based method.Complementary cases were those where only one team member made a correct prediction.
  • Limitations: The method relies heavily on the accuracy of approximated human decision-making models, and its complementary-case improvement was not very large.The authors attribute the limited improvement partly to limited human-AI complementarity.

4 PHASE II: COMMUNICATING HUMAN’S AND AI’S CORRECTNESS LIKELIHOOD TO PROMOTE APPROPRIATE TRUST

Phase II explored how to communicate and use modeled human-AI correctness likelihood during AI-assisted decision-making. It proposed three strategies and evaluated their effects on trust appropriateness and team performance.

  • Phase II design: The authors proposed three CL exploitation strategies: Direct Display, Adaptive Workflow, and Adaptive Recommendation.
  • Phase II design: A between-subjects experiment examined how different CL exploitation strategies affect human trust appropriateness and team performance.

humans’ perceptions and user experiences in the decision process?

The study compared five interface conditions for AI-assisted income prediction, varying whether participants received AI confidence, human-AI CL, adaptive sequencing, or selective recommendations. Participants completed training, independent and interactive task stages, assisted decisions, and an exit survey.

  • Experimental conditions: Direct Display presented estimated human and AI CL alongside the AI recommendation for the final human decision.The interface showed the two CL values side by side, followed by the AI’s predicted income.
  • Experimental conditions: Adaptive Workflow changed the decision order according to estimated CL: humans decided first when human CL was higher, otherwise AI advice appeared first.Participants made the final decision after reviewing the AI recommendation in both cases.
  • Experimental conditions: Adaptive Recommendation displayed the AI explanation by default but revealed the recommendation only when estimated AI CL exceeded human CL.When human CL was higher, the recommendation was hidden while the explanation remained available.
  • Study procedure: The between-subjects study recruited 300 participants and obtained 293 complete submissions, with 289 valid responses after attention-check filtering.The final condition counts were 59, 59, 59, 56, and 56, respectively.
  • Study procedure: Participants completed a tutorial, two 20-task batches, interactive decision-rule creation, condition-specific assisted decisions, and an exit survey.The survey collected demographics, subjective measures, and open-ended feedback about decision-making and interface designs.
  • Experimental conditions: The five conditions were Human Only, AI Confidence, Direct Display, Adaptive Workflow, and Adaptive Recommendation.All interfaces used a profile area on the left and a decision-making area on the right.

4.3 Evaluation Metrics

The evaluation measured team outcomes, human-AI trust alignment, perceived CL, subjective experiences, and open-ended perceptions using mixed quantitative and qualitative analyses.

  • Objective measures: Human-AI Agreement measured the fraction of tasks in which participants’ final decisions matched the AI recommendation, whether correct or wrong.
  • Objective measures: Team Performance measured the accuracy of participants’ final decisions.Participants also reported which side—human, AI, or both—they believed had higher CL for each task instance.
  • Subjective measures: The exit survey used 7-point Likert questions covering trust, decision confidence, complexity, mental demand, autonomy, satisfaction, future use, and CL-related perceptions.CL-related items included trust, usefulness, helpfulness for deciding when to trust AI, and acceptance of human CL estimation.
  • Subjective measures: The study also collected open-ended feedback about how participants used and perceived communicated human-AI CL and how interface designs affected decision-making.
  • Analysis: Nonparametric tests handled quantitative comparisons, while two authors analyzed open-ended feedback through inductive thematic coding.Pairwise tests included Mann-Whitney U or Wilcoxon tests; multi-group comparisons used Kruskal-Wallis tests with Bonferroni correction.

5 RESULTS

Across the experiment, exploiting both human and AI correctness likelihood promoted more appropriate trust and generally improved team performance over relying on AI confidence alone. Effects varied by CL relationship and whether AI confidence matched recommendation correctness.

  • Overall results: The three CL exploitation conditions achieved complementary performance, whereas AI Confidence did not outperform Human Only.
  • Trust appropriateness: The proposed conditions reduced agreement with wrong AI recommendations relative to AI Confidence, while agreement with correct recommendations did not differ significantly.Lower agreement with AI was better when the AI recommendation was wrong.
  • CL situations: Team performance was significantly better when human CL was higher than AI CL (p<.001 in all conditions).This pattern held when AI recommendations were either correct or wrong.
  • Confidence consistency: Adaptive Workflow and Adaptive Recommendation performed better than AI Confidence in the Consistent region, particularly in Low & Wrong cases.AI Confidence was marginally lower than Adaptive Workflow (p=.074) and significantly lower than Adaptive Recommendation (p<.001) overall.
  • Additional findings: Participants effectively received the estimated human-AI CL information and performed better on task instances where perceived and communicated CL were consistent.
  • Overall conclusion: Overall, the three strategies promoted appropriate trust by reducing over-trust without causing under-trust, contributing to better team performance.

6 DISCUSSION

The discussion argues that modeling and communicating both human and AI correctness likelihood can improve trust calibration, while identifying important usability, complementarity, measurement, and generalizability limits.

  • Human perceptions: Human self-confidence affects whether people follow AI advice, yet confidence in their own capabilities can mismatch actual capabilities.This mismatch can lead participants to overlook correct AI recommendations.
  • Human perceptions: Communicating numerical probabilities is difficult for people, motivating systems that compare human and AI correctness likelihood automatically.The proposed approach leaves probability estimation and comparison to the system and adapts the decision-making process or interface.
  • Complementary performance: The proposed three conditions exceeded AI Only and Human Only in team performance, but improvement was only about 3-4%.The limited gain was attributed to a relatively small complementary region between human and AI knowledge spaces.
  • Complementary performance: Human capability modeling can support richer assistance by estimating both people’s correctness likelihood and predictions.These estimates could help identify likely human-AI agreement and tailor interventions, such as encouraging analytical thinking when both may be wrong.
  • Limitations: Adaptive Recommendation sometimes induced skepticism, potentially hindering use of correct AI advice, and the method’s generalization to other domains and populations remains uncertain.The authors caution against broad generalization and note that users’ decision models can change during real-world interaction.
  • Limitations: The method’s scope is constrained by rule-based decision models, non-expert participants, low-stakes tasks, static short-term models, and trust measured through human-AI agreement.Agreement cannot distinguish following AI advice from independently reaching the same judgment.

7 CONCLUSION

The paper addresses appropriate trust in AI by modeling human correctness likelihood and combining it with AI capability information at the task-instance level. It proposes three exploitation methods and reports more appropriate trust than the traditional AI confidence-based approach.

  • Motivation: Appropriate trust in AI is framed as a central challenge in AI-assisted decision-making.The paper focuses on considering mutual capability information from humans and AI.
  • Method: The first phase approximates human decision-making models to estimate correctness likelihood on individual task instances.The method includes an interactive decision-rule modification interface.
  • Method: The second phase uses human-AI capability information to promote appropriate trust through three correctness-likelihood exploitation methods.The study examines effects on trust appropriateness, task performance, and user experience.
  • Findings: The proposed human correctness-likelihood modeling and exploitation method promoted more appropriate human trust in AI than the traditional AI confidence-based method.This is the paper’s principal reported conclusion.
Loading 2301.05809v1…