Conceptual Metaphors Impact Perceptions of Human-AI Collaboration

1 INTRODUCTION

• Similar conversational AI systems can receive sharply different user responses, which mental-model and cultural explanations alone do not fully explain.The introduction contrasts Xiaoice and Tay, which shared underlying Microsoft technology, and notes differing evaluations of Replika and Mitsuku within the same cultural context.Source: §1 · p. 2
• Conceptual metaphors are brief system descriptions that shape expectations beyond functional understanding and can apply across AI systems and tasks.The paper proposes that users evaluate identical interactions differently depending on whether a metaphor leads them to expect greater competence or warmth.Source: §1 · pp. 2–3
• The study uses the Stereotype Content Model’s warmth and competence dimensions to select metaphors and frame their expected effects on users.Crowdsourced labels place examples such as a toddler in the high-warmth, low-competence quadrant and a shrewd executive in the low-warmth, high-competence quadrant.Source: §1 · p. 3
• In a Wizard-of-Oz experiment with 260 participants, the researchers varied the agent metaphor while measuring warmth and competence expectations, usability, adoption intention, and cooperation desire.Participants saw the metaphor before completing a travel-planning conversation with a wizard who was blind to condition and randomized across participants.Source: §1 · p. 3
• Low-competence metaphors improved perceived usability, intention to adopt, and desire to cooperate, while high-warmth metaphors also prompted more positive views and longer interaction.The effects occurred despite wizard-operated agents performing at a human level; the authors interpret the competence pattern through Contrast Theory and the warmth pattern through Assimilation Theory.Source: §1 · p. 3
• The authors urge designers to scrutinize competence expectations in AI metaphors and compare their findings with user responses to past agents.The paper also outlines a finer-grained competence study and a study of users’ initial interest in trying systems; its retrospective analysis finds patterns consistent with adoption and cooperation for agents including Xiaoice, Tay, and Mitsuku.Source: §1 · pp. 3–4

2 RELATED WORK

• Pre-use expectations color evaluations of identical experiences and persist over time, while AI capability mismatches can limit use, motivating causal study of metaphor effects.Performance metrics may not create accurate expectations, so users develop folk theories; prior interviews found users retreat to menial tasks after encountering capability limits but did not establish causality or quantify the effect.Source: §2 · p. 4

2.1 Metaphors shape expectations

• Metaphors communicate complex system ideas, aid users in predicting agent behavior, and shape AI folk theories before use, yet their effects on evaluations were unstudied.Metaphors can describe human roles or other contexts, and designers use them to communicate an agent’s capabilities and boundaries.Source: §2.1 · pp. 4–5

2.2 Competing predictions: assimilation vs. contrast

• Assimilation theory predicts positive (high-competence or high-warmth) metaphors improve adoption intent and cooperation (H1), while contrast theory predicts they reduce both when systems fall short (H2).Assimilation aligns evaluation with initial expectations; contrast emphasizes the gap between expectations and actual experience, including disappointment when conversational agents reveal their limits.Source: §2.2 · pp. 5–6

3.1 Collaborative AI task

• Participants compared travel options and chose a vacation package meeting specified requirements across hotel, outbound-flight, and return-flight goals.The prompt asked participants traveling alone from Montreal to consider New York, Berlin, or Paris between August 23 and September 1; they were also asked to consider factors such as Wi-Fi, breakfast, and spa options.Source: §3.1 · pp. 6–7

3.2 Wizard-of-Oz conversational agent

• A Wizard-of-Oz agent supported the task, with wizards blind to treatment and using a finite travel database to constrain responses.The database contained hotel and flight information, and wizards were instructed to say when requested information was unavailable.Source: §3.2 · p. 7

3.3 Sampling metaphors

• Study 1 used four metaphors, one per warmth and competence quadrant, chosen after pretesting each with 50 mutually exclusive Amazon Mechanical Turk workers rating conversational-agent warmth and competence on a 5-point Likert scale.Source: §3.3 · p. 8

4 STUDY 1: METAPHORS DRIVE EVALUATIONS

• The study examines metaphor effects on pre- and post-use usability and warmth, post-use adoption intention and cooperation, and behavioral differences in chat logs.Source: §4 · p. 9

4.1 Procedure

• Participants took part in a between-subjects experiment with four metaphor conditions varying in warmth and competence, plus a no-metaphor control.Source: §4.1 · p. 9
• After consenting, participants were introduced to one of the four metaphors or the no-metaphor control condition.Source: §4.1 · p. 9
• Before interacting with the system, participants reported their expectations of its competence and warmth.Source: §4.1 · p. 9
• Participants then received a goal-oriented task and chatted with a wizard posing as the agent until completing it.Source: §4.1 · p. 9
• After finalizing travel plans, participants evaluated the system, answered a manipulation check, and were debriefed that they had interacted with a human.Source: §4.1 · p. 9

User evaluation measures

• Pre-use expectations and post-use perceptions of competence and warmth were measured separately, without asking whether expectations were met or violated, to avoid making contrast salient.The questions used a 5-point Likert scale, and question ordering was randomized.Source: §4.2 · pp. 9–10
• Usability indices were α = 0.91 pre-use and α = 0.87 post-use; warmth indices were α = 0.94 pre-use and α = 0.91 post-use, drawing on the stereotype content model.Usability was measured with adapted survey items; warmth items asked whether the system was good-natured and warm.Source: §4.2 · p. 10
• After interaction, two-item indices measured adoption intentions (α = 0.98) and cooperation and helping intentions (α = 0.79), drawing on prior HRI work.Adoption items asked about willingness to continue using the service and likelihood of future use; cooperation items asked about cooperating with or helping the AI.Source: §4.2 · p. 10

Conversational behavior measures

• LIWC’s 82 language dimensions were compared across participant words, wizard words, and combined conversations; word counts and interaction duration were also measured at message and conversation levels.These measures were used to examine differences in conversational behavior across conditions.Source: §4.2 · p. 10

4.3 Participants

• Across the studies, participants were US citizens aged 18 or older recruited through Amazon Mechanical Turk, and each could participate only once.Responses were discarded if participants failed to follow instructions, left the conversation midway, or did not follow task specifications.Source: §4.3 · p. 11
• Participants received $4 for a survey averaging 15 ± 5 minutes, roughly $15 per hour; 50.7% were female and mean age was 41 ± 10.3.Compensation followed fair-work standards on Mechanical Turk.Source: §4.3 · p. 11
• 140 participants remained after exclusions, exceeding the power-analysis target of 125 participants (25 per condition).The analysis assumed a small effect size of f = 0.05, significance level of 0.05, and 80% power.Source: §4.3 · p. 11

4.4 Wizard-of-Oz manipulation check

• The end-of-survey manipulation check asked how participants thought the system worked technically, without explicitly directing attention to whether it might be human.The approach drew on prior Wizard-of-Oz studies.Source: §4.4 · p. 11
• Two coders individually reviewed responses and flagged participants who suspected a person was pretending to be an AI, excluding flagged participants from further analysis.Responses describing vocabulary parsing, training, or a database of answers were examples of participants’ explanations of how the system might work.Source: §4.4 · p. 11
• 13 participants (10.4%) were identified by both coders (κ = 1) as suspecting a human wizard and excluded.In most cases, suspicion was triggered by a wizard making a typo or taking too long to respond.Source: §4.4 · p. 11

4.5 Results: Metaphors shape pre-use expectations

• Before use, low-competence metaphors lowered expected usability relative to high-competence and control metaphors; warmth had no main effect.The low-competence mean was 2.41 ± 0.89, versus 3.10 ± 0.86 for high competence and 2.99 ± 1.03 for control.Source: §4.5 · pp. 11–12, Figure 3 · p. 12
• Before use, low-warmth metaphors lowered expected warmth relative to high-warmth and control metaphors; competence had no main effect.The low-warmth mean was 2.71 ± 1.12, versus 3.53 ± 1.05 for high warmth and 3.35 ± 1.23 for control.Source: Figure 3 · p. 12, §4.5 · p. 12
• Participants generally expected conversational AI to be highly competent and warm, while metaphor priming affected those expectations and reduced their variance.The figure indicates that high-competence and high-warmth metaphors corresponded to higher pre-use usability and warmth expectations, respectively.Source: §4.5 · p. 12, Figure 3 · p. 12

4.6 Results: Metaphors impact user evaluations and user attitudes

• Low-competence metaphors yielded higher post-use usability (4.02 ± 0.72 vs. 3.58 ± 1.07) and adoption intention (4.25 ± 0.96 vs. 3.42 ± 1.33) than high-competence metaphors.Competence had a main effect on both outcomes; the authors interpret the adoption result as support for contrast theory, with users more likely to adopt agents that outperform low expectations.Source: §4.6 · pp. 12–13, Figure 4 · p. 13
• Desire to cooperate was higher with low-competence and high-warmth metaphors, with competence and warmth both showing main effects and no interaction.The reported means were 4.39 ± 0.67 for low competence versus 3.69 ± 0.93 for high competence, and 4.23 ± 0.81 for high warmth versus 3.85 ± 0.92 for low warmth.Source: Figure 4 · p. 13, §4.6 · pp. 13–14
• Post-use warmth showed no significant main effects of competence or warmth and no interaction, although ratings were numerically higher in the high-warmth condition.The high- and low-warmth means were 3.64 ± 1.17 and 3.50 ± 1.12, respectively; pre-to-post warmth changed significantly only in the low-warmth condition.Source: §4.6 · p. 13

• LIWC analysis found no significant language-level differences across conditions (p > .05), suggesting evaluations reflected metaphor-set expectations rather than conversation content.The wizard was blinded to condition, though LIWC may miss language shifts it cannot capture.Source: §4.7 · p. 15
• High warmth increased words per conversation from 82 ± 37 to 101 ± 45 (F(1, 106) = 5.35, p < 0.05, η2 = 0.05).Source: §4.7 · p. 15
• Participants in the high-warmth condition spent an average of 4 ± 1.5 minutes longer interacting, and qualitatively asked more questions and explored the system’s capabilities.In the example, a participant testing a low-competence, high-warmth bot continued to report high intention to adopt and cooperate despite its failures.Source: §4.7 · p. 15

4.8 Summary

• Users tolerate knowledge gaps in low-competence agents more than mistakes by high-competence agents, supporting contrast theory for competence.Intention to adopt and desire to cooperate also decrease as the metaphor’s projected competence increases.Source: §4.8 · p. 15
• High-warmth agents prompt users to cooperate more and interact longer, providing some support for assimilation theory.The passage reports no significant impact of warmth on users’ intention to adopt.Source: §4.8 · p. 15

5.1 Procedure

• The five primed metaphors ranged from “toddler” to “trained professional,” lay in the high-warmth half of the space, and were supplemented by a no-metaphor control group.After using the system, control participants selected which of the five metaphors they felt described it most accurately.Source: §5.1 · pp. 15–16

5.2 Measures

• Across the five metaphors, the study measured participants’ intention to adopt and desire to cooperate.Source: §5.2 · p. 16

5.3 Participants

• The study recruited 120 AMT participants, 20 per condition; their average age was 39±11.06 and 48% identified as female.Source: §5.3 · p. 16

5.4 Wizard-of-oz manipulation check

• The two coders agreed on the same five suspicious participants (κ = 1), a 4.1% suspicion rate, and those participants were removed from analysis.Source: §5.4 · p. 16

5.6 Summary

• Even with human-level performance in every condition, users were more likely to adopt a lower-competence agent; underperforming expectations reduced adoption further.The benefit was asymmetric: exceeding extremely low expectations helped more than exceeding slightly higher, still-low expectations.Source: §5.6 · p. 17

6.1 Procedure

• Unlike earlier experiments, participants rated likelihood of trying an agent described by a Study 1 metaphor without interacting with the agent or a wizard.The between-subjects experiment presented each participant with one metaphor and asked about trying a new AI service.Source: §6.1 · p. 17

6.2 Measures

• The trial index combined likelihood of trying the system and envisioning long-term use (α = 0.94), while the cooperation index combined willingness to cooperate and tolerate errors (α = 0.78).Both indices were formed from two questions rated on 5-point Likert scales.Source: §6.2 · pp. 17–18

6.3 Participants

• The survey included 80 new AMT participants, with 20 exposed to a metaphor from each warmth-competence quadrant; none had participated in the other studies.Source: §6.3 · p. 18

6.4 More interest in trying out high competence and high warmth AI systems

• High-competence, high-warmth metaphors produced the highest trial index (3.35 ± 1.06), compared with 1.63 ± 0.84 for low competence and high warmth.Overall trial-index means were 2.95 ± 1.18 for high competence versus 1.5 ± 0.83 for low competence, and 2.48 ± 1.28 for high warmth versus 1.96 ± 1.16 for low warmth. Competence (F(1, 100) = 42.96, p < 0.001, η2 = 0.34) and warmth (F(1, 100) = 5.63, p = 0.020, η2 = 0.04) had significant effects; the reported interaction was F(1, 100) = 1.54, p = 0.021, η2 = 0.012.Source: §6.4 · p. 18
• High-competence, high-warmth metaphors produced the highest pre-use cooperation index (3.40 ± 0.75), compared with 1.65 ± 0.82 for low competence and high warmth.Overall cooperation-index means were 2.80 ± 1.11 for high competence versus 1.58 ± 0.84 for low competence, and 2.53 ± 1.18 for high warmth versus 1.85 ± 1.04 for low warmth. Competence (F(1, 100) = 37.36, p < 0.001, η2 = 0.28), warmth (F(1, 100) = 5.63, p = 0.01, η2 = 0.08), and their interaction (F(1, 100) = 6.86, p = 0.001, η2 = 0.052) were reported as significant.Source: §6.4 · p. 18

6.5 Summary

• The study found that high competence may attract prospective users, while metaphor choices must also support favorable evaluations and cooperative behavior after use.The authors contrast this pre-use benefit with earlier studies’ detrimental effects from presenting an AI system with a high-competence metaphor, and report that high warmth also increased likelihood of trying a service and behaving positively with it.Source: §6 · p. 17, §6.5 · p. 18

7 DISCUSSION

• Metaphors can set expectations for algorithmic systems beyond conversational agents and AI, although their implications may vary with task, interaction, and context.The discussion gives news feeds, content curation, and recommender systems as examples of systems people reason about through metaphors.Source: §7 · pp. 18–19

7.1 User behavior around algorithmic systems

• Where prior work examined users’ metaphorical folk theories and personification of algorithmic systems, this work studied how metaphors explicitly attached by designers affect experience.The discussion places this approach alongside HCI research on how people reason about algorithmic systems and interact with them.Source: §7.1 · p. 19

7.2 Design implications

• Studies 1 and 2 found that low-competence metaphors produced the highest agent evaluations, while Study 3 found they were least likely to be tried; high warmth was beneficial, so designers could lower or gradually raise projected competence after attracting users.The proposed approaches include tempering expectations when interaction begins or introducing an agent as a trainee that may evolve into a professional.Source: §7.2 · p. 19

7.3 Limitations and future work

• The study examined a text-only conversational AI in structured 15–20-minute interactions, so future work should test visual metaphors, embodied agents, open-ended conversation, prolonged exposure, and the role of prior technology experience.The structured task did not produce significant vocabulary differences across conditions, despite differing evaluations.Source: §7.3 · p. 20
• Participants preferred cooperating with agents projecting higher warmth but still perceived a gap between projected and actual warmth, prompting calls for stronger warmth theories and tests of more extreme warmth violations.Further work should also examine trustworthiness, task failure, and how metaphors affect evaluations when the system’s actual competence or warmth is varied below the study’s human-level performance.Source: §7.3 · p. 20

7.4 Retrospective analysis

• The five chatbot descriptions measured signaled high competence, between “recent graduate” and “trained professional,” while Tay also projected very low warmth.The authors suggest such descriptions may set systems up for failure, but note that expectations cannot be shown to have caused these specific reception patterns.Source: §7.4 · pp. 21–22, Figure 7 · p. 21
• Xiaoice’s high-warmth framing contrasted with Tay’s high-competence, low-warmth framing, alongside Xiaoice’s continued popularity and Tay’s removal after 16 hours; Mitsuku received antisocial messages in up to 30% of cases.Woebot and Replika also combined high warmth with comparable competence expectations to Tay’s and garnered users; the authors caution that other variables affect reception and metaphors alone cannot be credited.Source: §7.4 · pp. 21–22

8 CONCLUSION

• Conceptual metaphors shape users’ pre-use expectations and post-use evaluations, adoption intentions, and desire to cooperate with AI agents.Source: §8 · p. 22
• People are more likely to cooperate with agents they expect to be warm, and more likely to adopt and cooperate with agents projecting low competence.Source: §8 · p. 22
• This low-competence advantage runs counter to designers’ usual default of projecting high competence to attract users.Source: §8 · p. 22