Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies

INTRODUCTION

• The study tests chatbot identity and inquiry strategies in a persuasive conversation, where users must infer the bot’s intent and assess its messages against their own ideas.The authors use the CASA framework and Uncanny Valley of Mind theory to guide their investigation.Source: INTRODUCTION · pp. 1–2
• An online 2 × 4 factorial experiment recruited 790 participants to test Jessie versus Jessie [bot] and four inquiry strategies in a children’s-charity donation task.The study used an agenda-based persuasive dialogue system built with the PERSUASIONFORGOOD dataset.Source: INTRODUCTION · p. 2
• Perceived human identity was associated with greater donation likelihood and higher competence, confidence, sincerity, and warmth ratings, with the donation effect especially salient under personal inquiry.Participants who perceived a chatbot displayed as Jessie as a bot had the lowest donation likelihood, while those who perceived it as human had the highest.Source: INTRODUCTION · p. 2

Effect of chatbot identity on donation

• The study frames perceived chatbot identity as consequential for persuasion and partner impressions, contrasting with the CASA-based expectation of similar outcomes for human and machine identities.The authors also connect identity disclosure and intentions to ethical considerations in persuasive chatbot design.Source: INTRODUCTION · p. 2, Effect of chatbot identity on donation · p. 2

Persuasive Inquiries’ Effect on Donation

• Prior work links asking about a behavior with subsequent behavior change, including exercise and blood donation, motivating inquiry as a possible influence on donation.The authors note that this question-behavior effect may be particularly salient for socially normative behaviors such as donation.Source: Persuasive Inquiries’ Effect on Donation · p. 2
• The authors predict that personal inquiries, alone or paired with non-personal inquiries, will yield better persuasive and interpersonal outcomes than non-personal inquiries or no inquiry.The rationale is that personal questions can tailor information, arguments, or feedback to a persuadee.Source: Persuasive Inquiries’ Effect on Donation · p. 2

Interaction between Identity and Inquiries on Donation

• The authors hypothesize an identity-by-inquiry interaction, reasoning that a human-perceived chatbot using personal, human-like inquiries may evoke uncanniness and lower impressions and persuasiveness.They invoke Uncanny Valley of Mind theory, including the possibility that a human-seeming partner with machine-like qualities can elicit discomfort.Source: Interaction between Identity and… · pp. 2–3

Persuasive Strategies

• The adopted PERSUASIONFORGOOD scheme includes ten strategies: seven appeals to influence donation and three inquiries intended to support personalization and closer relationships.The appeals include credibility, emotional, logical, self-modeling, foot-in-the-door, personal-story, and donation-information strategies.Source: Persuasive Strategies · p. 3
• Non-personal inquiries ask about the organization or donation task, while personal inquiries ask for relevant personal information, such as whether the persuadee has children.Table 1 presents templates for the two inquiry types.Source: Persuasive Strategies · p. 3, Table 1 · p. 3

PERSUASIVE DIALOGUE SYSTEM

• The agenda-based system adapts human-human PERSUASIONFORGOOD conversations into a human-bot setting using NLU, a dialogue manager, and NLG modules.The NLU interprets user dialogue acts, the dialogue manager selects the next system act, and NLG turns it into a human-readable response.Source: PERSUASIVE DIALOGUE SYSTEM · p. 3

Natural Language Understanding (NLU)

• The NLU module maps user utterances to one of 14 predefined dialogue acts and uses utterance, textual, contextual, and character features for classification.The classifier reached 62% test accuracy; adding regular expressions and predefined rules raised accuracy to 84.1%.Source: Natural Language Understanding (NLU) · pp. 3–4

Dialogue Manager (DM)

• The dialogue manager follows a fixed agenda: it begins with persuasive inquiry, then presents appeals in a controlled order, with donation information following the first donation proposal.The appeal order is credibility, emotional, logical, self-modeling, foot-in-the-door, and personal story.Source: Dialogue Manager (DM) · p. 4
• When a user agrees to donate, the system obtains or confirms the amount, proposes a larger donation, and then closes the conversation.If the user does not agree, the system closes after exhausting its appeals; factual questions can route to the Factual QA module and then return to the prior agenda step.Source: Dialogue Manager (DM) · pp. 4–5

Natural Language Generation (NLG)

• Inquiry messages use fixed templates so their surface forms remain controlled across experiments, isolating the effects of inquiry strategy.The paper distinguishes template-based, retrieval-based, and generation-based NLG approaches.Source: Natural Language Generation (NLG) · p. 5
• Persuasive appeals use retrieval rather than templates to support more coherent, varied responses across conversational contexts while keeping the NLG module shared across groups.The system retrieves from PERSUASIONFORGOOD because its candidate messages are considered sufficient and its data size limits generation capacity.Source: Natural Language Generation (NLG) · p. 5
• For appeals, the system represents the last user utterance with TF-IDF-weighted bag-of-words vectors, ranks candidate contexts by similarity, and randomly selects among the top three.This procedure is intended to avoid repetitive responses and encourage diversity.Source: Natural Language Generation (NLG) · p. 5

THE PERSUASION TASK

• Participants spoke with a persuasive chatbot about Save the Children, with any donation deducted from their $2 task payment and privately confirmed after the conversation.The task included pre-task and post-task surveys; the confirmed amount could differ from what participants promised in conversation.Source: THE PERSUASION TASK · p. 5
• Researchers recruited 790 participants through Amazon Mechanical Turk; the task took about 15 minutes, and each participant received $2 after completion.ParlAI was used for data collection, and study procedures were approved by the authors’ institutions.Source: THE PERSUASION TASK · pp. 5–6

EXPERIMENT DESIGN AND METHODS

• The 2 × 4 factorial experiment varied identity disclosure (“Jessie” or “Jessie (bot)”) and four inquiry strategies, from no inquiry to both personal and non-personal inquiries.The identity label appeared at the beginning of every system utterance, and inquiry conditions used two template questions when applicable.Source: EXPERIMENT DESIGN AND METHODS · p. 6

Participant Demographics

• The 790 participants were distributed approximately 100 per condition; demographic distributions did not significantly differ across conditions, supporting successful randomization.Participants varied in gender, age, education, ethnicity, and bot familiarity.Source: Participant Demographics · p. 6

Dependent Variables

• Donation probability measured whether a participant donated, expressed as the group’s relative donation frequency; donation amount was dichotomized as 1 for donation and 0 for no donation.The authors describe donation probability as a more important persuasion-success indicator than amount, which was capped at $2.Source: Dependent Variables · p. 6
• Partner impressions were measured on five-level ordinal scales for competence, confidence, warmth, and sincerity.Source: Dependent Variables · p. 6
• Conversation quality included five-level ratings of engagement, response naturalness, and perceived persuasiveness.Source: Dependent Variables · p. 6

Measurement Validation

• The KMO value was 0.78, above the recommended 0.6, and Bartlett’s test was significant (χ2(561) = 9,639.27, p < 0.001).The authors report these tests as indicating the data were sufficient for further analysis.Source: Measurement Validation · p. 6

RESULTS

• Bonferroni correction was applied to p-values from the multiple t-tests reported in the results.The results are presented in five areas, including donation, partner impression and conversation quality, LIWC features, inconsistent donation behavior, and personality.Source: RESULTS · p. 6

Persuasion Outcome - Donation Behavior

• Across conditions, estimated donation probability averaged 45%, versus 54% in earlier human-human persuasion research; no-inquiry conditions were higher, but not significantly.Figure 2 presents donation probabilities across displayed identities and inquiry strategies on the whole dataset.Source: Persuasion Outcome - Donation Behavior · p. 6, Figure 2 · p. 7
• Perceiving the chatbot as human was associated with higher donation probability, and the positive effect of personal inquiries versus no inquiry was stronger for participants who perceived a human partner.Personal inquiries were not effective when participants perceived the partner as a bot; the perceived-identity interaction was significant.Source: Figure 3 · p. 7, Persuasion Outcome - Donation Behavior · p. 7
• Among participants who believed the displayed identity, displaying “Jessie (bot)” decreased donation probability (β = −0.52, p < 0.05).With personal inquiry, donation probability was significantly lower for “Jessie (bot)” than for “Jessie” (p < 0.05); the identity-by-inquiry interaction in logistic regression was not significant.Source: Figure 4 · p. 7, Persuasion Outcome - Donation Behavior · p. 8
• Displayed identity and suspicion interacted: participants shown “Jessie” but perceiving a bot were least likely to donate, while those shown “Jessie” and perceiving a human were most likely.Adding inquiry revealed a significant three-way interaction among inquiry, displayed identity, and suspicion (p < 0.05).Source: Persuasion Outcome - Donation Behavior · pp. 7–8

Partner Impression and Conversation Quality

• The “Jessie (bot)” label was associated with higher perceived response naturalness (β = 0.18, p < 0.05), despite similar message quality across displayed-identity conditions.The authors suggest participants may have lower expectations when the bot directly discloses its identity.Source: Partner Impression and Conversation… · p. 8
• Participants perceiving the partner as human rated it higher in competence, confidence, sincerity, and warmth, and rated conversation quality higher across engagement, naturalness, and persuasiveness (p < 0.001).Figure 6 showed many significant results, whereas Figure 5 did not.Source: Partner Impression and Conversation… · p. 8, Figure 5 · p. 9
• In the reduced dataset, the “Jessie (bot)” identity had a significantly negative impact on all four partner impressions and all three conversation-quality metrics (p < 0.001).The authors used reduced-dataset analyses in the following sections to provide a clearer picture of the results.Source: Partner Impression and Conversation… · p. 8
• Competent impressions, natural responses, and persuasiveness each contributed positively to perceiving the chatbot as human (p < 0.05).The model used the reduced dataset and treated perceived identity as its outcome.Source: Partner Impression and Conversation… · p. 8
• Among participants who perceived the partner as a bot, greater competence was associated with higher donation probability (β = 0.53, p < 0.01).The authors also report significant positive correlations between donation outcome and competent and warm impressions.Source: Partner Impression and Conversation… · pp. 8–9, Table 5 · p. 9

LIWC Linguistics Features

• Participants used more positive emotional words when they perceived the partner as human and more distinction-related words, such as “but” and “not,” when they perceived a chatbot (p < 0.001).The authors suggest users may express feelings more with a perceived human and make more distinctions with a perceived chatbot.Source: LIWC Linguistics Features · p. 9
• The study used LIWC (Linguistic Inquiry and Word Count) to analyze clue words from different linguistic categories across perceived identities.The cited analysis describes these categories as linguistic features of participants’ conversations.Source: LIWC Linguistics Features · p. 9
• The authors interpret the distinction-related words as suggesting that users feel a greater need to make distinctions when they perceive the partner as a chatbot.Examples include “but” and “not.”Source: LIWC Linguistics Features · p. 9

Inconsistent Donation Behavior

• Participants who perceived the partner as human were more likely to make inconsistent donations (β = 0.70, p < 0.01).Inconsistent donation meant the actual amount differed from the amount indicated during conversation.Source: Inconsistent Donation Behavior · p. 9
• For participants perceiving a human, personal inquiry with the human “Jessie” identity was associated with more consistent donations than the no-inquiries condition (β = −1.52, p < 0.05).The authors suggest personal inquiry may shorten interpersonal distance and encourage consistency.Source: Inconsistent Donation Behavior · p. 9
• More rational decision-making style and stronger endorsement of purity were associated with keeping donation promises (p < 0.05).These personality measures predicted consistency, not donation amount.Source: Inconsistent Donation Behavior · p. 10

Donation Behavior and Personality

• Being male (β = 0.62, p < 0.05), having medium income (β = 0.81, p < 0.01), and having the “caring” trait (β = 0.42, p < 0.05) predicted donation.Controlling for these variables left the main donation, identity, and inquiry findings consistent with earlier results.Source: Donation Behavior and Personality · p. 10

ETHICAL CONSIDERATIONS

• The authors argue that bot identity disclosure is necessary, noting that some participants perceived an identified bot as human and that disclosure may elicit reactance to personal inquiries.They describe this reactance as potentially protective against abusive persuasive-chatbot uses and call for further study.Source: ETHICAL CONSIDERATIONS · p. 10
• The authors call for persuasive chatbots to benefit users and society through accurate information, transparency, and user autonomy, with greater caution for contentious health or political goals.They state that defining ethical boundaries for those contexts is beyond this paper’s scope.Source: ETHICAL CONSIDERATIONS · p. 10

LIMITATIONS AND FUTURE WORK

• The study’s findings are bounded by a rigid agenda and template- or retrieval-based responses, a simple “Jessie (bot)” identity cue, conversations limited to about 10 turns, and an online donation context.The authors recommend testing longer conversations and other persuasion domains.Source: LIMITATIONS AND FUTURE WORK · p. 10

CONCLUSIONS

• In the 790-participant donation study, perceived identity—not displayed identity—impacted donation outcomes; the authors also report that personal inquiry worked better for people who thought they were talking to a human.They identify chatbot response naturalness as a quality to improve and describe their code, data, and analyses as a first step toward persuasive dialogue systems.Source: CONCLUSIONS · p. 10

• Figure 7 shows “Jessie” and “Jessie (bot)” before system utterances to mark chatbot identities across experimental conditions.Source: Figure 7 · p. 14
• Figure 8 presents the chat interface.Source: Figure 8 · p. 14
• Table 6 provides example sentences for seven persuasive appeals from [46].Source: Table 6 · p. 15
• Figure 9 depicts interaction effects between suspicion and displayed identity.Source: Figure 9 · p. 15
• Figure 10 depicts three-way interaction effects among suspicion, displayed identity, and persuasive inquiry.Source: Figure 10 · p. 15