PaperlayerView exact revision on arXiv

Source-linked AI summary

The Placebo Effect of Artificial Intelligence in Human-Computer Interaction

Thomas Kosch, Robin Welsch, Lewis Chuang, Albrecht Schmidt

arXiv:2204.04935v1cs.HC

TL;DR

The paper asks whether system descriptions can bias evaluations of adaptive systems. Across two experiments, it examines expectancy effects and finds that descriptions influence subjective performance expectations and ratings, but not objective task performance.

  • Problem

    HCI has given little systematic attention to placebo effects on user expectations, despite their potential to bias evaluations of adaptive systems.

    Source: §2.2 · p. 5

  • Method

    Two experiments manipulate system descriptions to examine placebo effects on expectations, subjective ratings, and task performance.

    Source: §6 · p. 19, §7 · p. 24

  • Results

    Expectations persisted after interaction, but there was no direct effect of manipulated expectations on task performance across the two studies.

    Source: §6 · p. 19

  • Takeaways & Limitations

    HCI evaluations of AI and other systems should consider placebo-controls when system descriptions may shape user expectations.

    Source: §6 · p. 19, §7 · p. 25

    The study did not control for individual differences such as verbal fluency or stress, which could have affected task performance across conditions.

    Source: §6.4.6 · p. 24

Abstract

from arXiv · show

In medicine, patients can obtain real benefits from a sham treatment. These benefits are known as the placebo effect. We report two experiments (Experiment I: N=369; Experiment II: N=100) demonstrating a placebo effect in adaptive interfaces. Participants were asked to solve word puzzles while being supported by no system or an adaptive AI interface. All participants experienced the same word puzzle difficulty and had no support from an AI throughout the experiments. Our results showed that the belief of receiving adaptive AI support increases expectations regarding the participant's own task performance, sustained after interaction. These expectations were positively correlated to performance, as indicated by the number of solved word puzzles. We integrate our findings into technological acceptance theories and discuss implications for the future assessment of AI-based user interfaces and novel technologies. We argue that system descriptions can elicit placebo effects through user expectations biasing the results of user-centered studies.

1 INTRODUCTION

Placebo effects in HCI can arise when system descriptions shape expectations, potentially biasing evaluations of novel interfaces. This work investigates whether descriptions of adaptive AI alter subjective evaluations and task performance.

1 INTRODUCTION

  • Unlike medical and other human-response research, placebo-controls are not the norm when evaluating novel human-machine interfaces.Medical studies use placebo-controls so treatment effects can be distinguished from improvements attributable to expectations.Source: §1 · pp. 1–2
  • HCI placebo effects may occur when prior expectations lead users to evaluate a system favorably, even without active system support.Earlier work showed effects on immersion and satisfaction, but the authors note a lack of evidence on personal effectiveness or objective task performance.Source: §1 · p. 2
  • The experiments compared descriptions of adaptive AI with a No Adaptivity condition while participants completed word puzzles.The described AI was said to adapt puzzle difficulty using webcam physiology or task errors, but participants always experienced No Adaptivity.Source: §1 · p. 3
  • Expectations persisted after interaction, but neither experiment found a statistically significant effect on error rates.Experiment II found a placebo effect on subjective performance ratings for Physiology-based Adaptation.Source: §1 · p. 3

CONTRIBUTION STATEMENT

The paper demonstrates an HCI placebo effect in two word-puzzle experiments and discusses implications for evaluating AI and other systems. It also considers how expectations can be assessed and controlled in experimental designs.

CONTRIBUTION STATEMENT

  • Three system descriptions suggesting improvement in subjective performance ratings were presented to participants in a word-puzzle task.The ratings were collected after interaction.Source: CONTRIBUTION STATEMENT · p. 3
  • The paper reports demonstrating a placebo effect in HCI through two experiments inspired by a literature search on placebo elements in HCI studies.Source: CONTRIBUTION STATEMENT · p. 3
  • The paper discusses implications for evaluating AI and other systems and ways to assess and control expectations in between- and within-subject designs.Source: CONTRIBUTION STATEMENT · p. 3

2 RELATED WORK

The related work covers user expectations, adaptive-interface assessment, and prior HCI studies of placebo effects. It motivates a systematic investigation of how expectations may affect evaluations of adaptive systems.

2 RELATED WORK

  • The review surveys standard usability and acceptance measures, adaptive AI interfaces, and research on placebo effects in HCI.The literature search identified 27 publications from an initial set of 420 ACM Digital Library results.Source: §2 · p. 4, §2.3 · p. 5

2.1 User Expectations

  • Prior work links expectations about technology to satisfaction and evaluation, while leaving open how suggested but nonfunctional adaptation affects expectations.Earlier work found that informing players about functional game adaptation could change playing performance, but did not test a suggested, nonfunctional placebo.Source: §2.1 · p. 4

2.2 Usability and Assessment of Adaptive User Interfaces

  • Subjective usability measures can be biased, and descriptions of nonfunctional adaptive systems can increase satisfaction or performance ratings.The review notes that objective measures are less affected by subjective biases.Source: §2.2 · pp. 4–5

2.3 A Placebo Effect in User-Centered Studies

  • The literature search distinguishes studies that discuss placebo effects, use placebo-controls, or compare sham treatments against a baseline.The review describes examples in which sham system descriptions affected user evaluations and operationalizes placebo effects through changes in measured metrics.Source: §2.3 · pp. 5–6

3 EVALUATING PLACEBO EFFECTS OF AI IN USER STUDIES

The study tests whether descriptions of adaptive AI alter user expectations, subjective performance ratings, and objective performance. Two experiments use word puzzles with task difficulty held constant.

3 EVALUATING PLACEBO EFFECTS OF AI IN USER STUDIES

  • Participants completed word puzzles while system descriptions manipulated expectations about alleged difficulty adaptation; puzzle difficulty never changed.The experiments were conducted remotely through a browser-based web app.Source: §3 · p. 6
  • No Adaptivity and Error-based Adaptation were described as transparent and low-complexity, while Physiology-based Adaptation was low-transparency and high-complexity.These properties were assigned to the experimental conditions to manipulate participants’ expectations.Source: Table 2 · p. 7
  • The research question asks whether descriptions of adaptive systems confound their evaluation, including expectations, performance ratings, task performance, and workload.The hypotheses propose that descriptions can manipulate expectations and performance measures, and reduce perceived workload.Source: §3 · p. 7

3.1 General Study Approach

  • Experiment I used a between-subject design; Experiment II let each participant compare Physiology-based Adaptation with No Adaptivity.The first design avoided cross-condition probing, while the second enabled within-person comparison of the alleged system and baseline.Source: §3.1 · p. 7

4 EXPERIMENT I: EXPLORING THE PLACEBO EFFECT OF AI

Experiment I examined whether descriptions of adaptive AI influence performance expectations and task outcomes, including ratings, accuracy, errors, and workload. It also assessed links between expectations and performance and motivated a within-subject replication to address individual differences.

4.1 Experimental Design and System Description

  • Participants were randomly assigned to No Adaptivity, traceable Error-based Adaptation, or complex Physiology-based Adaptation, but everyone received the same puzzles without actual difficulty adjustment.The error-based description said difficulty would respond to correct and incorrect answers; the physiology-based description invoked webcam-derived heart rate and facial-emotion measures, while assuring participants video was neither recorded nor transmitted.Source: §4.1 · pp. 7–8, Error-based Adaptation · p. 8

4.1.1 Task Description and Apparatus.

  • Participants solved 20 randomized word puzzles with 45-second limits; the study measured errors, response time, expected performance, and post-task workload, then debriefed participants about the deception.After observing limited group variation for three- and eight-character words, the authors used validated five-character puzzles for Experiment II.Source: §4.1.1 · p. 9, §4.1.2 · p. 9, §4.1.3 · p. 10

4.2 Participants

  • Of 369 participants, 255 remained after preprocessing exclusions for failed comprehension checks, low puzzle accuracy, rapid completion, or incomplete responses.Source: §4.2 · p. 10

4.5 Number of Errors

  • Accuracy averaged 69.90%, and conditions did not differ significantly; pre-task expectations correlated with correct responses overall (r = .25, p < .001), most strongly in Physiology-based Adaptation (r = .40, p < .001).Post-task performance ratings also correlated with accuracy (r = .40, p < .001), uniformly across conditions (r values .39–.44).Source: §4.5 · pp. 12–13

4.6 Workload

  • Workload did not differ significantly across conditions; higher workload was associated with lower post-task performance ratings (r = -.24, p < .001) and accuracy (r = -.35, p < .001).Pre-task performance expectations were unrelated to workload (r = -.01, p = .818), and the workload–accuracy relationship was largely uniform across groups.Source: §4.6 · p. 13

4.7 Specificity of Perceived Adaptiveness

  • Participants perceived adaptiveness as group-specific: both adaptive conditions were rated as adapting to task performance, but only Physiology-based Adaptation was perceived as adapting to stress.Error-based Adaptation received higher task-performance adaptiveness ratings than Physiology-based Adaptation, whereas the physiology condition received higher stress-based adaptiveness ratings.Source: §4.7 · p. 14, §4.10 · p. 15

4.8 Technology Acceptance

  • Most technology-acceptance scales differed between adaptive conditions; performance expectancy and attitude were slightly higher for Error-based Adaptation, and the authors omitted UTAUT from Experiment II.Source: §4.8 · p. 14

4.4 Subjective Evaluation

  • Error-based Adaptation increased expected performance before the task and perceived performance afterward versus No Adaptivity; neither comparison was significant for Physiology-based Adaptation.The pattern of higher performance ratings for Error-based Adaptation held across both time points; ratings declined after participants solved the puzzles overall.Source: §4.4 · pp. 11–12, Figure 4 · p. 12, §4.10 · p. 14

4.10 Discussion

  • No effects were found for physiology-based adaptation on errors (H2) or perceived workload (H3), while expectation variance predicted errors, especially in that condition.The researchers chose a within-subject replication to control for individual perceptions and variance in beliefs about adaptivity.Source: §4.10 · p. 15

5 EXPERIMENT II: A PLACEBO EFFECT WHEN COMPARING INTERFACES

Experiment II tested whether describing an interface as adaptive changes expectations and task outcomes when participants compare it with a No Adaptivity condition. Expectations favored the described adaptive interface and remained higher after interaction, but objective performance and workload did not differ significantly.

5.1 Method

  • The study used a counterbalanced within-subjects comparison of No Adaptivity and Physiology-based Adaptation, with 40 puzzles per condition and calibration and expectancy measures.Participants solved five practice puzzles before each condition; before each quiz, they estimated how many of 40 puzzles they would solve correctly.Source: §5 · p. 16, §5.1 · p. 16

5.2.1 Participants.

  • After data cleaning, 75 of the 100 recruited participants remained for analysis; exclusions covered 19 participants who failed all three validation questions and 10 instances of completion below 10%.Source: §5.2.1 · p. 17

5.2.2 Subjective Evaluation.

  • Participants expected better performance with Physiology-based Adaptation than with No Adaptivity, and expectations remained distinguishable after interaction despite an overall decrease.The adaptivity effect was significant, F(1, 222) = 74.70, p < .001; expectations decreased by about 1.5 points from before to after system exploration.Source: §5.2.2 · pp. 17–18, Figure 8 · p. 18

5.2.3 Number of Errors.

  • Accuracy did not differ significantly between conditions (61.20% Physiology-based Adaptation vs 60.77% control), while expectation differences correlated with accuracy (r(73)=.28, p=.014).The condition comparison was t(74) = -0.15, p = .882, d = -0.0212.Source: §5.2.3 · pp. 18–19

5 EXPERIMENT II: A PLACEBO EFFECT WHEN COMPARING INTERFACES

  • Expectations favored Physiology-based Adaptation, but objective performance did not differ; expectation differences correlated with performance (r(73)=.28, p=.014).The reported accuracy means were 61.20% for Physiology-based Adaptation and 60.77% for the control group.Source: §5.2.3 · pp. 18–19, §5.2.4 · p. 19

5.2.4 Workload.

  • NASA-TLX workload did not differ significantly between conditions (p = .074), although it was descriptively higher with Physiology-based Adaptation.Mean workload was 71.79 with Physiology-based Adaptation and 66.83 in the control condition.Source: §5.2.4 · p. 19

6 DISCUSSION

The discussion distinguishes subjective expectancy effects from direct task-performance effects, while noting an association between expectations and performance. It connects these findings to technology acceptance and recommends controlling for expectations in HCI evaluations.

6 DISCUSSION

  • Descriptions changed performance expectancy and common usability ratings, but no direct task-performance effect emerged; expectancy was associated with correct-answer gains.The authors report that participants who believed more in the Physiology-based Adaptation's superiority showed larger performance gains.Source: §6 · p. 19

6.1.1 Technology Acceptance.

  • Expectations should join usability in technology-acceptance models: judgments persisted after use, while performance gains appeared only among those expecting them.The authors note that demographics and prior experience may mitigate both performance expectations and their influence on user behavior.Source: §6.1.1 · p. 20

6.1.2 Placebo Theory and Classification.

  • Because the adaptive AI was novel to participants, the findings align with expectancy-based placebo accounts and add digital artifacts as a placebo category.The authors distinguish this from conditioned-response accounts, which rely on previously learned stimulus-response associations.Source: §6.1.2 · p. 20

6.2 Controlling for Placebo Effects

  • Because HCI users may recognize novel interfaces and hold nonneutral expectations, the authors recommend polling before and after use and using control groups or statistical adjustment.They note that placebo-control strategies may be inefficient, unethical, or infeasible in some settings, and that proposed alternatives need further investigation.Source: §6.2 · p. 20, §6.2.1 · p. 21, §6.2.2 · p. 21

6.3.1 Persistence of Effects.

  • Expectation-linked performance gains and sustained subjective beliefs were clearest for physiology-based adaptation; inaccurate framing may backfire when expectations are disconfirmed.The authors report that the trend was not clear for Error-based Adaptation and that post-interaction performance-expectancy ratings favored it over Physiology-based Adaptation.Source: §6.3.1 · p. 22, §6.3.2 · p. 22

6.3.3 FATE: Fairness, Accountability, Transparency, and Explainability.

  • A semi-accurate mental model may support expectancy confirmation, while beliefs in nonfunctional AI capabilities may also raise overtrust risks in safety-critical settings.The authors suggest exposing AI workings and testing post-hoc explanations; they also discuss potential accountability shifts alongside automation bias.Source: §6.3.3 · pp. 22–23

6.4.1 Assessing Qualitative Data.

  • Future work should probe mechanisms with mixed methods, incentivize careful responding, test prolonged use, and add nocebo conditions.The authors also propose examining whether qualitative interviews and expectation-aware analyses can help address evaluation bias.Source: §6.2.3 · p. 22, §6.4.1 · p. 23, §6.4.2 · p. 23

6.4.7 Generalizability to other HCI Studies.

  • Findings are bounded to implicit AI conditions and participants believing in the assistant's superiority; individual traits were uncontrolled and the factors behind description effects remain unresolved.Perceived novelty did not differ between adaptive conditions, but the authors recommend modeling individual variation in follow-up work.Source: §6.4.7 · p. 24, §6.4.6 · p. 24, §6.4.5 · p. 24

7 CONCLUSION

The paper reports a placebo effect in HCI: system descriptions can affect expectations and task performance concerning alleged technological innovation. It argues that HCI evaluations should consider placebo control.

7 CONCLUSION

  • System descriptions can affect subjective expectations and task performance concerning alleged technological innovation.The authors report a placebo effect in HCI when participants collaborate with an AI.Source: §7 · p. 24
  • The authors argue that HCI studies should consider placebo control, as in other human-related research fields.They provide the application source code, collected data, and analysis scripts to support further research.Source: §7 · p. 25

APPENDIX

The appendix introduces the seven-point Likert scale used for Likert items.

APPENDIX

A VALIDATION QUESTIONS

Participants answered validation questions before solving the puzzles to check whether they understood the instructions. Those who did not answer all three correctly were excluded from analysis.

A VALIDATION QUESTIONS

  • Participants answered three validation questions before solving puzzles; those who missed any answer were excluded from analysis.Source: A VALIDATION QUESTIONS · p. 25
  • The questions tested participants’ understanding of the task-difficulty conditions and what the AI analyzed.The listed correct responses identify no adaptation versus AI-based camera analysis, and pulse and emotional states inferred from facial features.Source: A VALIDATION QUESTIONS · p. 25

B CUSTOMIZED QUESTIONS

The appendix lists customized questions asked after each condition, alongside adapted UTAUT questions. The custom items address stress adaptation, puzzle difficulty, and prior use of similar systems.

B CUSTOMIZED QUESTIONS

  • After each condition, participants answered custom seven-point Likert questions about stress adaptation, puzzle difficulty, and regular use of similar systems.Source: B CUSTOMIZED QUESTIONS · p. 25
  • Table 4 presents adapted UTAUT questions asked after every condition.Source: Table 4 · p. 26

C ADAPTED QUESTIONS

The study adapted five seven-point Likert questions to fit the task narrative and assess whether participants perceived the word puzzles as stress-adaptive.

C ADAPTED QUESTIONS

  • Five of six questions from Denisova et al. were adapted to the task narrative and presented on seven-point Likert scales.The omitted question measured system intervention on a player's character.Source: C ADAPTED QUESTIONS · p. 26
  • The items ask whether puzzles were generated, appeared, or matched according to the participant's stress level.They also ask whether puzzle behavior changed with boredom or stress and whether adaptation was sensible.Source: C ADAPTED QUESTIONS · p. 26

D UTAUT

The study measured user expectancy and satisfaction with adapted UTAUT items after each condition, and also administered Wells et al.'s perceived-novelty questions. Both sets used seven-point Likert items.

D UTAUT

  • Three UTAUT items—Performance Expectancy, Effort Expectancy, and Hedonic Motivation—measured user expectancy and satisfaction after each condition.The questions were adapted to the study objectives; Table 4 presents them.Source: D UTAUT · p. 26

E WELLS ET AL.

  • All six Wells et al. items on perceived novelty and technology adoption were administered after each condition using seven-point Likert items.The questions are presented in Table 5.Source: E WELLS ET AL · p. 26

F NASA-TLX

Perceived task load for each adaptivity was measured with the raw NASA-TLX, calculated from six items on a 1–20 scale.

F NASA-TLX

  • Raw NASA-TLX scores quantified perceived task load for each adaptivity using six items rated from 1 to 20.The scale endpoints are Very Low and Very High.Source: F NASA-TLX · p. 26
  • The six NASA-TLX dimensions were mental, physical, and temporal demand; performance; effort; and frustration.The prompts ask about task demands, accomplishment, work required, and feelings such as insecurity or irritation.Source: F NASA-TLX · pp. 26–27
Loading 2204.04935v1…