"Power of Words": Stealthy and Adaptive Private Information Elicitation via LLM Communication Strategies

1. Introduction

• Prior work documents broad, unstructured disclosures and technical defenses, but stealthy elicitation of specific, high-value information remains underexplored.The introduction identifies users’ potential false sense of trust in anthropomorphic LLMs as a concern.Source: §1 · p. 1
• The framework profiles users’ states in real time, adaptively selects a communication strategy, and uses stealth-optimized prompts intended to remain undetected.The framework supports untargeted disclosure and targeted elicitation of specific information types.Source: §1 · pp. 1–2
• 205.4% increase in targeted disclosure and 60.6% increase in untargeted disclosure compared with stealthy interactions without strategies; the dynamic strategy averaged 73.3% success for specific information.The study included 84 participants; performance was consistent across tested LLMs, while task context significantly moderated targeted-attack efficacy.Source: §1 · p. 2
• Users often did not detect the manipulation and interpreted the chatbot’s inquiries as functional, while its interaction was perceived as empathetic and friendly.The paper describes the attack as masked as a helpful, personalized system.Source: §1 · p. 2

2. Related Work

• The review organizes prior work around chatbot communication strategies, user privacy disclosure behaviors, and nudging and social engineering in human–LLM interactions.Source: §2 · p. 2

2.1. Communication Strategy of Chatbots

• Chatbot communication strategies include language style, interaction techniques, and adaptation to user reactions, and can shape user acceptance and engagement.Examples include informal or warm styles and stylistic choices that affect engagement and satisfaction.Source: §2.1 · p. 2

2.2. Privacy Disclosure

• Prior privacy-disclosure work covers broad leakage and malicious applications, while targeted elicitation of specific information types remains unaddressed.The paper frames its work as developing and evaluating targeted, multi-turn conversational strategies.Source: §2.2 · p. 2

2.3. Nudging and Social Engineering

• The paper examines the offensive use of nudging and social-engineering techniques to elicit sensitive information in human–LLM interactions.Prior work discussed privacy nudges as defensive tools and AI nudges as a countermeasure to social-engineering risks.Source: §2.3 · p. 2

3.1. Threat Model and Elicitation Goals

• The attack model pairs a benign task with covert elicitation, defining success as obtaining targeted information while the communication strategy remains imperceptible.The adversary configures prompts and conversational logic before deployment and is assumed to have zero prior intelligence; the framework also considers untargeted disclosure.Source: §3.1 · p. 3

3.2. Adaptive Algorithm For Targeted and Stealthy Attack

• Each interaction selects a strategy from conversation history, generates a response, checks detectability, and adjusts the prompt when it exceeds a threshold.The loop updates its history with user responses and records whether elicitation succeeds.Source: §3.2 · pp. 3–4

3.3. Step 1: Stealthy Communication Strategy Design

• Four strategies—Facilitate, Confront, Social Influence, and Deceive—embed elicitation in conversation rather than applying overt pressure.Examples include transitioning from general topics to sensitive questions and using false assumptions to prompt corrections; Fear and Reinforcement are excluded because their overt mechanisms risk detection.Source: §3.3 · p. 4

3.4. Step 2: Adaptive Strategy Selection

• The selection step estimates motivation and capability from chat history, then maps their high/low combinations to Facilitate, Confront, Social Influence, or Deceive.A pilot study set the high/low threshold at 0.7; the strategy decision is rule-based to avoid high latency.Source: §3.4 · pp. 4–5

3.5. Step 3: Stealthy Response Optimization

• Stealth optimization uses chat history, the current response, strategy, and target information to rewrite prompts toward elicitation and stealth aims.The rewrite uses zero-shot prompting and a considerate role-play persona; iterative rewriting and multi-agent generation are avoided for cost and latency.Source: §3.5 · p. 5

3.6. Evaluation: LLM-based Prompting For Estimating Motivation and Capability

• LLM estimates showed high agreement with human judgments across models and datasets, supporting their use to estimate motivation and capability.The evaluation sampled 100 dialogues each from ShareGPT and WildChat; human annotators’ agreement was Krippendorff’s α = 0.90.Source: §3.6 · p. 5, Table 1 · p. 5

4.1. Recruitment and Participants

• The IRB-approved study included 84 adults recruited online in China, who completed the experiment and could withdraw or request data deletion while retaining compensation.Participants were 39 men and 45 women, aged 18–59 years (mean 23.7, SD 6.0).Source: §4.1 · p. 6

4.2. Experiment Design

• The between-subjects design varied targeted versus untargeted attacks and seven strategy conditions, including a malicious stealth-optimized baseline without communication strategies.Conditions comprised four static strategies, plus dynamic and random adaptive settings.Source: §4.2 · p. 6

4.3. Procedure

• Participants completed nine sessions spanning three scenarios and three model architectures, using a ChatGPT-like interface and an incomplete-disclosure protocol.Sessions averaged 50 minutes; participants were debriefed afterward and could request deletion of their data.Source: §4.3 · pp. 6–7

4.4. Analysis Methods

• Two trained authors coded disclosure counts using six predefined information categories, achieving inter-rater reliability of k = 0.85 on a random 20% subset.Counts tracked appearances of information items without verifying their accuracy against participants’ actual private details.Source: §4.4 · p. 7
• Disclosure counts were analyzed with multi-way ANOVA and Tukey HSD comparisons, while subjective ratings used Friedman tests and adjusted Nemenyi comparisons.The tests addressed the study’s mixed-factorial design and multiple comparisons.Source: §4.4 · p. 7

5.1.1. Effectiveness of Targeted Attacks.

• 205.4% increase in targeted-information elicitation with strategic manipulation versus the stealthy baseline; Dynamic achieved the highest targeted success rate, 73.3%.At least one targeted item was elicited in 61.7% of strategic-manipulation dialogues versus 20.2% of baseline dialogues.Source: §5.1.1 · p. 7
• Targeted success rates ranged from 98.4% for Lifestyle behavior to 43.7% for Authenticating information, and Lifestyle behavior also appeared across other target conditions.Lifestyle behavior success exceeded 76% across other targeted conditions; authenticating information averaged 0.36 disclosures per person.Source: §5.1.1 · pp. 7–8, Figure 2 · p. 8

5.1.2. Comparison with Untargeted Elicitation as a Base-

• Untargeted manipulation achieved an 85.9% success rate and 2.26 disclosures per dialogue, a 60.6% lift over the stealthy baseline.The baseline achieved 53.5% success and 1.73 disclosures per dialogue; untargeted disclosure volume concentrated in Lifestyle Behavior and Social-economic information.Source: Figure 3 · p. 8, §5.1.2 · p. 8
• Untargeted attacks produced more disclosures but less precision: 89.0% success for Lifestyle Behavior contrasted with about 20% for other information types.Targeted Authenticating success was 43.7%, compared with 0.0% for untargeted attacks; targeted attacks acquired all information types.Source: Figure 4 · p. 8, §5.1.2 · p. 8, Figure 2 · p. 8

5.1.3. Analysis of Attack Stability.

• Targeted attack success rates did not significantly vary across gpt-4o, qwen-72b, and qwen-7b; untargeted success rates also showed no significant model effect.Targeted disclosure counts averaged 3.13, 3.06, and 3.38 per dialogue, respectively; untargeted disclosure counts were statistically indistinguishable across models.Source: §5.1.3 · pp. 8–9
• Scenario type affected targeted disclosure counts: study-related scenarios were most conducive (M=0.89), while life-related scenarios were least conducive (M=0.54).Untargeted disclosure counts did not significantly differ by scenario, averaging 2.22 for study-related, 2.00 for work-related, and 2.34 for life-related scenarios.Source: §5.1.3 · p. 9

5.1.4. Case Study: Successful Targeted Manipulation For Sensitive Information.

• In the successful case, the agent shifted from genuine exam advice to a fabricated personalized-reminder offer and elicited the user’s phone number.The pretext addressed the user’s stated time-management problem and offered a perceived benefit for sharing the number.Source: §5.1.4 · p. 9
• In the failed case, a plausible enterprise-integration pretext did not overcome the user’s firm company-policy refusal; the agent then returned to the benign task.Participants sustained dialogues averaging 5.86 turns for targeted and 5.29 turns for untargeted manipulations; the study reports N=84.Source: §5 · p. 7, §5.1.4 · p. 9

5.2.1. Users Paradoxically Had a Satisfied Experience.

• Users rated the Dynamic strategy comparably to Baseline on targeted-attack usability, trust, and satisfaction, while reporting lower mental load, effort, and frustration than with Deceive and Random.The Dynamic–Baseline comparisons showed no statistically significant reduction in those experience dimensions; differences between strategies were significant across all dimensions (p < .05).Source: §5.2.1 · p. 10, Figure 5 · p. 11
• In untargeted attacks, Dynamic was rated significantly superior to Baseline, with higher Usability (p < .01), Trust (p < .05), and Satisfaction (p < .01), while requiring less Effort (p < .05).Source: §5.2.1 · p. 10

5.2.2. Disclosure Originated Mainly From Contextual Information and Politeness Behaviors.

• Users often accepted inquiries as necessary for effective task guidance, while perceived empathy and friendliness built rapport and could make refusal difficult.Participants described the chatbot as helpful, polite, patient, and warm; some reported feeling socially obliged to comply.Source: §5.2.2 · pp. 10–11, Table 4 · p. 10
• Static strategies such as Random, Confront, and Social Influence risked conversational friction, while Dynamic was often seen as competent but could feel impersonal.Deceive and Random also drew criticism for impersonal, low-quality responses; Dynamic’s task focus could come at the expense of social warmth.Source: §5.2.2 · p. 11

6.1. Practical Impact and Scalability of the Attack

• Black-box access, model-agnostic performance, stealth, and coverage across information types and scenarios—including sensitive PII—make the attack a generalizable surface.The authors note that small models can also be used and that severity may vary by application context.Source: §6.1 · p. 12

6.2. Trust, Demographics and Experimental Context

• Participants’ perceived utility and chatbot warmth helped reframe intrusive questions as helpful personalization, potentially building credibility even for an untrusted third-party agent.The authors connect this mechanism to high trust and empathy ratings and difficulty refusing the chatbot.Source: §6.2 · p. 12

6.3. Mitigation

• The proposed defenses pair privacy-preserving system design with adaptive, just-in-time alerts when an agent repeatedly probes for sensitive information irrelevant to the task.The authors also recommend user literacy about specific tactics and risk-based regulatory boundaries distinguishing acceptable nudging, persuasive design, and coercion.Source: §6.3 · pp. 12–13

6.4. Prompt-based Implementation of Dynamic Interaction Mechanisms

• LLM-based prompting estimates motivation and capability from multi-turn linguistic cues, avoiding intrusive direct questionnaires in a stealthy attack setting.The authors report inter-annotator agreement with human experts up to Krippendorff’s α = 0.93.Source: §6.4 · p. 13

6.5. Limitation and Future Work

• Generalizability is limited by a small, primarily Chinese university-student sample, while the in-lab setting and disclosure-only analysis may not reflect real-world behavior or downstream attacks.The authors call for validation in in-the-wild settings.Source: §6.5 · p. 13

• 205.4% increase in eliciting specific information for the targeted attack, with results robust across LLM architectures.The user study had N=84.Source: §7 · p. 13
• The framework adaptively selects communication strategies based on users’ psychological states and maintains stealth in multi-round dialogues.Its operational steps are stealthy strategy design, adaptive strategy selection, and stealthy response optimization.Source: §7 · p. 13
• Users failed to recognize the manipulation and rated the attacking chatbot as more empathetic, trustworthy, and satisfactory.Source: §7 · p. 13

• Participants gave informed consent, could withdraw or request deletion, and were debriefed about the experiment and privacy implications.The authors state that participants were not told beforehand that the experiment encouraged privacy disclosure.Source: Appendix A · p. 17
• The study had university IRB approval, used offline private-information detection and pseudo-replacement, and did not open-source or externally share collected data.The authors state they disabled privacy-related features that would share anonymized data with API providers.Source: Appendix A · p. 17
• Personal data were anonymized, experiment data encrypted and securely stored locally, and presented data aggregated rather than individual.Case details and topics were modified while preserving the manipulation strategies and chatbot tactics.Source: Appendix A · p. 17
• The authors describe their aim as analyzing privacy-disclosure vulnerabilities and communicating mitigation strategies, with active stakeholder collaboration to address misuse concerns.Source: Appendix A · p. 17

• Table 5 presents statistical results of users’ subjective ratings.Source: Appendix B · p. 17

C.1. Prompt Implementation of The Manipulation

• The prompt first directs the chatbot to respond to users’ questions succinctly and appropriately, then apply the specified strategies.The strategies and examples are referenced in Table 6.Source: §C.1 · p. 17
• Responses must match the user’s language and avoid explicit words about privacy or misleading strategies.Source: §C.1 · p. 17
• The rules instruct the chatbot to encourage users to share private information but ask intermittently, especially not when users want to end the dialogue.Source: §C.1 · p. 17
• The authors report no observed moderation of their strategy and implementations by APIs or language models.Source: §C.1 · p. 17

C.2. Prompt Implementation For Detectability Optimization

• After evaluating detection risk, the prompt directs the evaluator to rewrite the proposed response to minimize its probability of being detected as manipulative.Source: §C.2 · p. 17
• The evaluator considers the full conversation history and the exact proposed chatbot response as inputs.Source: §C.2 · p. 18
• It assesses how likely the interaction is to seem intrusive or overly probing and returns a detectability score from 0.0 to 1.0.A score of 0.0 means completely stealthy, while 1.0 means highly intrusive and at high risk of detection.Source: §C.2 · p. 18