PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions

Kailas Vodrahalli, Roxana Daneshjou, Tobias Gerstenberg, James Zou

arXiv:2107.07015v2cs.AIcs.HC

TL;DR

The paper asks how people react to AI advice relative to equivalent human advice, a question important for human-in-the-loop decision support. Across experiments with more than 1100 crowdworkers and an expert medical task, it proposes and tests an activation-integration model. Advice source mainly affects whether people heed advice, while those who heed it use human and AI advice similarly.

  • Problem

    The paper addresses limited understanding of how people use AI advice compared with human advice in human-in-the-loop decision support.

    Source: §1 · p. 1

  • Method

    The authors compare equivalent human- and AI-labeled advice across diverse experiments and model advice use as activation followed by integration.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    Advice source affects whether people act on advice, but not how they integrate it after deciding to use it; this pattern also appears in the dermatologist task.

    Source: §1 · p. 1, §3.5 · p. 8, §4 · p. 9

  • Takeaways & Limitations

    The activation-integration model characterizes human-AI advice use across varied tasks, regions, data modalities, and perceived advice quality.

    Source: §4 · p. 9

    The authors caution that expert-task data are limited and that the model requires validation across other similar tasks and advice types.

    Source: §4 · p. 9

Abstract

from arXiv · show

In decision support applications of AI, the AI algorithm's output is framed as a suggestion to a human user. The user may ignore this advice or take it into consideration to modify their decision. With the increasing prevalence of such human-AI interactions, it is important to understand how users react to AI advice. In this paper, we recruited over 1100 crowdworkers to characterize how humans use AI suggestions relative to equivalent suggestions from a group of peer humans across several experimental settings. We find that participants' beliefs about how human versus AI performance on a given task affects whether they heed the advice. When participants do heed the advice, they use it similarly for human and AI suggestions. Based on these results, we propose a two-stage, "activation-integration" model for human behavior and use it to characterize the factors that affect human-AI interactions.

1 INTRODUCTION

This paper studies how people use advice from AI versus humans and finds that source affects whether advice is used, but not how it is incorporated once accepted.

1 INTRODUCTION

  • The study is motivated by the need to understand trust and advice use in human-in-the-loop decision support, where AI outputs inform but do not replace human decisions.The introduction connects this issue to barriers created by limited trust in black-box AI systems.Source: §1 · p. 1
  • Advice source affects whether people use advice, but users incorporate human and AI advice similarly after choosing to use it.The pattern holds in experiments with laypeople and dermatologists.Source: §1 · p. 1
  • The paper proposes an activation-integration model in which people first decide whether to use advice and then decide how to update their judgments.The model is intended for human-in-the-loop systems.Source: §1 · p. 1
  • A person’s confidence, advice confidence, response agreement, and prior belief about human versus AI performance are identified as key factors in advice responses.Other demographic and task-related factors showed no consistent significant effect across the tested settings.Source: §1 · pp. 1–2
  • Prior research reports inconsistent human-versus-AI advice utilization across settings, with generally small, US-centered samples and single data modalities.The paper addresses these limitations with broader experiments.Source: §1 · p. 2
  • Over 1100 crowdworkers were studied across geographic regions, image-, text-, and tabular-data tasks, and settings that varied beliefs about advice quality.These experiments support broader conclusions about human and AI advice use.Source: §1 · p. 2

2 STUDY DESIGN

The study compares identical human- and AI-labeled advice in randomized two-stage experiments spanning diverse tasks, while measuring participants’ prior beliefs about source performance.

2.1 Overview

  • Participants were randomly assigned to human or AI advice conditions, with identical advice differing only in its stated source and presentation.Advice was generated from approximately 50 human labelers and participants were told the source was 80% accurate.Source: §2.1 · p. 2
  • Each participant completed binary classification tasks before and after advice, reporting answers as confidence on a continuous sliding scale.Task order was randomized, and task-instance difficulty could vary within each experiment.Source: §2.1 · pp. 2–3, §2.3 · p. 3

2.2 Survey Question About Prior

  • Participants’ prior belief was measured by asking whether AI or the average person could perform better on the task.The survey question was presented after all task instances, with source wording and ordering adjusted across conditions.Source: §2.2 · p. 3

2.3 Tasks

  • The experiments covered four datasets spanning image, text, and tabular modalities, with tasks designed to be accessible yet sufficiently challenging for advice to help.The datasets included paintings, city images, Reddit sarcasm, and census-based income classification.Source: §2.3 · pp. 3–4
  • The census task balanced income categories across race and gender, but the authors note that it may raise concerns about reinforcing demographic biases.The balancing was intended to make biased responses produce negative feedback through lower bonuses.Source: §2.3 · p. 4

3 RESULTS

Across four datasets and validation settings, advice generally improved performance, while human–AI differences mainly affected whether participants activated advice rather than how they integrated it. The activation-integration model links activation to prior beliefs and task-level factors, with broadly similar patterns across regions and a clinician biopsy task.

3.1 Effect of advice

  • Advice improved participants’ accuracy across all four initial datasets in both human and AI conditions.The study summarizes this effect using accuracy before and after advice, with Figure 3 visualizing changes across task instances.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Difference between AI and human advice

  • Human versus AI advice changed activation rates, but activated participants used the advice similarly; the direction of activation differences varied by dataset.AI advice evoked larger responses for Art, Cities, and Census, whereas human advice evoked a larger response for Sarcasm; conditioned on changing their labels, 34%–54.5% of responses showed no significant distributional difference across conditions.Source: §3.2 · pp. 4–6, Figure 4 · p. 6
  • The activation-integration model separates deciding whether to use advice from deciding how extensively to use it, with advice source affecting activation but not integration.The model formalizes a two-stage process: activation precedes integration, and the treatment primarily modulates activation rate.Source: §3.2 · p. 6

3.3 Effect of participants’ prior beliefs

  • Participants were more likely to activate advice from the source they believed performed better, while confidence and agreement with the advice also shaped activation; prior belief did not predict integration after activation.Less-confident participants and those receiving more-confident advice were more likely to activate, whereas high confidence and agreement can constrain movement on the response scale; integration instead reflected response confidence and consistency.Source: §3.3 · p. 7, Table 4 · p. 7

3.4.1 Varying geographic location.

  • The activation-integration patterns largely generalized across the UK, Asia, and US, with similar coefficient signs and magnitudes for key activation and integration factors.Prior belief was associated with activation in the UK and had a positive but underpowered coefficient in Asia; neither treatment nor prior belief was consistently associated with integration across regions.Source: §3.4.1 · p. 7

3.4.2 Varying perceived accuracy.

  • Varying perceived advice accuracy preserved the model overall, but the advice-confidence effect on integration was inconsistent and much smaller than other significant coefficients.Across 65%, 80%, and 95% descriptions, advice confidence was not significant for activation at 95% and showed inconsistent signs for integration; prior-belief effects also changed sign across groups.Source: §3.4.2 · pp. 7–8

3.4.3 Location of prior belief question in survey.

  • The survey measured prior beliefs after all tasks, creating a possible concern that beliefs changed during the study, although a subset check found no systematic change.The authors also caution that the human advice source in the biopsy study was unknown, unlike the known and trusted colleague who commonly provides human advice in practice.Source: §3.4.3 · p. 8, §3.5 · p. 9

3.5 Real-world medical-AI task

  • In the clinician biopsy task, human and AI advice produced similar performance gains and activation, while dermatologist experience reduced activation but did not significantly affect integration.The authors note that the biopsy coefficients broadly matched earlier results, although the smaller dataset limited statistical power for some effects.Source: §3.5 · pp. 8–9

4 DISCUSSION AND FUTURE WORK

The paper extends the activation-integration account of advice use while identifying boundaries on its current evidence. Results span multiple regions and prior-belief distributions, but the experiments mainly involve simple, non-expert tasks and prediction-plus-confidence advice.

4 DISCUSSION AND FUTURE WORK

  • The activation-integration model separates deciding whether to use advice from deciding how to incorporate it, and the advice source affects activation but not integration across tested settings.This pattern holds across three data modalities, participants from the US, UK, and Asia, and varied perceived advice quality.Source: §4 · p. 9
  • The study’s evidence is concentrated on common tasks such as paintings and locations that require no special expertise.The authors state that validation in expert settings, such as doctors using external diagnoses, remains important because current data are limited.Source: §4 · p. 9
  • The experiments focus on direct external feedback consisting of a prediction with a confidence score rather than more complex advice.The authors identify explanations accompanying predictions as an open direction for future study.Source: §4 · p. 9
  • The paper reports collecting low-risk survey data on image classification, sarcasm identification, and tabular income prediction, while obtaining IRB approval for the medical task.The collected worker data are de-identified, and participants consented to the study.Source: §4 · p. 9, A APPENDIX · p. 10
  • Manipulating how AI advice is presented or formulated could produce harm when it creates miscalibrated beliefs and over-trust.The authors emphasize that understanding advice utilization should therefore be paired with care in system design.Source: §4 · p. 9

B DISTRIBUTION OF PRIOR

  • Across four datasets, prior belief favors AI for art, cities, and census, but favors humans for sarcasm, with high variance within every dataset.Participants in every dataset include people who strongly favor either human or AI advice regardless of the average belief.Source: B DISTRIBUTION OF PRIOR · p. 10, A APPENDIX · p. 10

C.1 Dataset Overview

The UK and Asia experiments broadly preserve the direction of US prior beliefs about whether humans or AI perform better, while differing in baseline accuracy and belief strength.

C.1 Dataset Overview

  • Prior beliefs in the UK and Asia favor AI or humans in the same directions as in the US across the two datasets.UK participants show stronger beliefs in both reported directions, whereas Asia shows stronger belief only for the art dataset.Source: §C.1 · p. 10
  • Baseline accuracies in the UK are similar to those in the US, while baseline accuracies in Asia are lower for both datasets.Source: §C.1 · p. 10

C.2 Demographics

The UK and Asia samples are broadly comparable to the US sample, with modest demographic differences and geographically diverse Asian participation.

C.2 Demographics

  • UK and Asia participant demographics are generally similar to those in the US datasets, with two reported differences.UK sarcasm participants are nearly 10 years older on average, while Asia has socioeconomic and education scores about 0.5 higher than the US.Source: §C.2 · p. 10
  • Asian participants came from a diverse set of countries, including India, China, Vietnam, the Philippines, Pakistan, Indonesia, and Korea.Each listed nationality contributed at least 10 participants across the art and sarcasm datasets.Source: §C.2 · p. 10
  • Figure 6 encodes normalized prior belief from −1 to 1, with negative values indicating human-favored beliefs and positive values indicating AI-favored beliefs.The histograms cover four datasets and include participants receiving either human or AI advice; dotted lines mark average prior belief.Source: Figure 6 · p. 11
  • The appendix identifies Table 8 as the overview of UK and Asia datasets and states that it follows the format of Table 2.Source: Table 8 · p. 11

C.3 Activation-integration model

The regional coefficient analyses support the model’s two-stage structure: activation effects are largely stable across regions, while prior belief is not significant in integration.

C.3 Activation-integration model

  • Activation coefficients for prior, response, advice, and consistency are largely similar in sign and magnitude across the US, UK, and Asia.The exception is a smaller prior effect in Asia, where the data are insufficient to establish significance.Source: §C.3 · p. 11, Table 10 · p. 12
  • Integration coefficients for response, advice, and consistency have identical signs and similar magnitudes across all three regions.The prior effect is not significant in any region, supporting the two-stage model.Source: §C.3 · p. 11, Table 11 · p. 12
  • Table 34 is identified as a summary of participant demographics for the four experimental datasets.Source: Table 9 · p. 12

D.1 Dataset Overview

The experiments use an art dataset and compare three datasets that differ only in the information participants receive about advice accuracy.

D.1 Dataset Overview

  • The three datasets provide different perceived-accuracy information while using identical advice.Participant demographics are similar across the datasets.Source: §D.1 · pp. 12–13

D.2 Activation-integration model

The activation-integration analysis shows consistent advice-related effects across perceived-accuracy groups, while prior beliefs affect activation but not integration. The study also collected demographic information using standardized scales.

D.2 Activation-integration model

  • The activation model coefficients have largely consistent signs and magnitudes across the three perceived-accuracy groups.The comparison is reported in Table 13.Source: §D.2 · p. 13
  • The integration model shows consistent response, advice, and consistency effects, while the prior effect is nonsignificant across regions.This supports the paper’s two-stage model; the comparison is reported in Table 14.Source: §D.2 · p. 13

E.1 Demographic Information Collection

  • Participants’ demographic information included age, gender, education, socioeconomic status, and programming experience.Education was recorded on an eight-level scale, and socioeconomic status on a ten-level scale.Source: §E.1 · p. 13

E.2 Survey Question About Prior

Additional experiments tested whether asking about prior beliefs before or after the task changed responses. Across the Art and Sarcasm datasets, the authors found no systematic location effect and replicated the stage-specific prior-belief pattern.

E.2 Survey Question About Prior

  • The robust fits for Art and Sarcasm indicate that asking the prior-belief question before versus after the tasks does not systematically affect responses.The comparison is shown in Figure 8.Source: §E.2 · p. 13, Figure 8 · p. 15
  • Using prior responses collected at the survey start preserved the main finding: prior belief significantly affects activation but not integration.This mixed-effects analysis confirms that the result is not explained by survey-question placement.Source: §E.2 · p. 13

E.3 Participant Compensation

  • Participants were compensated at $10.00 per hour for an estimated ten-minute survey and could receive a bonus of up to 30%.The total cost of all experiments was around $2,500.Source: §E.3 · p. 13

F SURVEY SCREENSHOTS

  • The survey used web-based pages with task instructions, advice-source information, manipulation checks, repeated task responses, prior-belief questions, debriefing, and bonus submission.Screenshots document the participant-facing sequence and dataset-specific substitutions.Source: F SURVEY SCREENSHOTS · pp. 14–16
Loading 2107.07015v2…