Source-linked AI summary
"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents
Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, Tianshi Li
TL;DR
LLM-based conversational agents raise privacy concerns, while prior research has largely lacked a user-centered account of disclosure risks and perceptions. The paper analyzes real-world ChatGPT conversations and interviews 19 users. It finds that users balance privacy against utility and convenience, while flawed mental models, dark patterns, and human-like interaction complicate that process.
Problem
Prior research primarily examined technical LLM privacy risks, leaving users’ disclosure behaviors and risk perceptions insufficiently understood.
Method
The study qualitatively analyzed real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based conversational-agent users.
Results
Users balanced privacy against utility and convenience, while erroneous mental models, dark patterns, and human-like interactions complicated privacy-risk navigation.
Takeaways & Limitations
Privacy-preserving LLM applications should improve users’ awareness and perceived and actual control through user-facing privacy-preserving techniques.
Takeaways & Limitations
The study focused primarily on ChatGPT, limiting understanding of privacy concerns on platforms with different interaction styles or user demographics.
Abstract
from arXiv · showhide
The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users' perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users' erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users' ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigm shifts to protect the privacy of LLM-based CA users.
1 INTRODUCTION
LLM-based conversational agents create privacy risks as users disclose sensitive information in high-stakes and open-ended interactions. This study examines users’ disclosure behaviors, perceptions, and trade-offs through conversation analysis and interviews.
- Motivation: LLM-based conversational agents in healthcare, finance, and counseling often require users to disclose private records or personal experiences.These disclosures may reach both hosting organizations and third-party application builders, creating privacy and security risks.
- Motivation: Open-ended, human-like interactions can increase both opportunities for sensitive disclosure and the scale of associated privacy risks.Compared with older, more constrained conversational agents, LLM-based systems invite broader and deeper personal disclosures.
- Research gap: Prior research largely measured or mitigated technical privacy risks, leaving users’ disclosure behaviors and risk perceptions comparatively underexamined.The paper addresses this gap with a human-centered investigation of how users navigate concrete benefits against more abstract privacy risks.
- Approach: The authors analyzed 200 real-world ChatGPT sessions containing 10,380 messages and interviewed 19 ChatGPT users across three research questions.The dataset study examined disclosure behaviors, while interviews addressed motivations, trade-offs, mental models, and privacy controls.
- Key findings: Participants used ad-hoc privacy protections while perceiving trade-offs among privacy, convenience, and utility.Disclosure intentions were primarily affected by perceived AI capability, and participants often believed they could not maximize privacy and utility simultaneously.
- Key findings: Erroneous mental models and dark patterns limited users’ understanding and exercise of privacy controls.Most participants did not know they could opt out of model training, while the opt-out interface linked privacy protection with perceived utility loss.
2 BACKGROUND AND RELATED WORK
LLM-based conversational agents combine memorization risks with human-like interaction risks that can encourage broader disclosure. Existing privacy research remains predominantly model-centered, motivating user-facing and mental-model research.
- Privacy risks: LLM-based conversational agents pose traditional privacy risks and LLM-specific memorization risks.Sensitive training data may be memorized and later exposed in generated outputs.
- Human-like interaction: Natural-language interaction can make agents seem human-like, increasing trust, perceived competence, and private information disclosure.The paper connects human-like interaction with gradually increasing sensitive disclosure and nudging of what users share.
- Existing methods: Model-centered privacy techniques include data sanitization, differentially private training, and post-training knowledge unlearning.These approaches primarily address privacy during training or after model development rather than users’ interaction decisions.
- Research gap: User-facing privacy-preserving research for LLM applications remains limited and lacks comprehensive user-perspective evidence.Even ProPILE, a privacy-leakage probing tool, did not include need-finding research or user evaluation.
- Related disclosure research: Online disclosure research identifies perceived benefits, costs, social influence, and platform trust as factors shaping disclosure decisions.This paper extends that context to LLM-based agents because their interactive communication may produce broader and deeper disclosure.
- Mental models: Inaccurate mental models can produce unsafe use and inappropriate trust, making them relevant to privacy risks in LLM-based agents.Mental-model research examines how users understand system components, relationships, and possible privacy threats.
3 DATASET ANALYSIS
The dataset analysis examined real-world ChatGPT disclosures using automated PII detection, qualitative coding, and a multidimensional scenario typology. It found diverse disclosures involving users’ own and other people’s information, including sensitive experiences.
- Dataset: The ShareGPT52K dataset contains 50,496 ChatGPT histories collected from December 2022 through March 2023.Each history includes user prompts and ChatGPT responses, with examples of names and personal experiences.
- Sampling: Microsoft Presidio identified PII in 30K conversations, which were divided into 7K higher-density and 23K lower-density groups.The groups were distinguished by whether conversations averaged more or less than one detected PII per turn.
- Sampling: The final qualitative sample contained 200 conversation threads with multiple turns, averaging 51.9 turns per conversation.Conversation lengths ranged from 2 to 572 turns.
- Analysis: The analysis used contextual integrity to examine actors, data types, and contexts of disclosure.Researchers relabeled detected PII at conversation level and created a tool to identify whose information appeared.
- Limitations: The dataset analysis was limited by unclear conversational context, unobservable user thought processes, and sample bias.ShareGPT extension users may not represent all LLM users, so the authors interpret observed behaviors conservatively.
- Findings: The sample included highly identifiable data, less directly identifiable information, and sensitive personal experiences without typical PII.Users also disclosed other people’s PII, creating interdependent as well as institutional privacy concerns.
- Typology: Researchers built a typology with four dimensions: Context, Topic, Purpose, and Prompt strategy.The dimensions characterize work, academic, or life settings; topics; user goals; and interaction tactics.
- Disclosure scenarios: Data Analysis conversations were particularly prone to sensitive disclosures because users often pasted data tables for interpretation.Interactive task definition also led users to gradually reveal more information across multiple rounds.
4 INTERVIEW METHODOLOGY
The interview study used semi-structured interviews with 19 LLM-based CA users to examine disclosure behaviors, privacy perceptions, mental models, and experiences with privacy controls. Researchers elicited participants’ own conversation examples, conducted mental-model activities, and qualitatively coded the resulting interviews.
- Participants: 19 LLM-based CA users participated, including 18 ChatGPT users and one Bing Chat user.Interviews were conducted remotely between July and August 2023 after IRB approval.
- Participants: Participants were recruited through Prolific and the authors’ social network using a prescreening survey covering CA experience, demographics, and technical background.Recruitment materials did not mention privacy, and scenario-based questions were used instead of directly asking what data participants had shared.
- Interview protocol: Participants prepared at least three personal conversations and discussed their goals, disclosures, and privacy concerns during interviews.They could redact information they did not want researchers to see.
- Interview protocol: Participants drew mental models of how LLM-based CAs use inputs to generate responses and improve services, then received a debriefing about system operation.The interview also addressed storage, training, and memorization risks.
- Interview protocol: The protocol examined awareness and experiences with ChatGPT privacy controls, including history, deletion, sharing, and training opt-out features.Researchers also asked about account sharing and plugin use; the Bing Chat participant was asked only about history deletion.
- Qualitative analysis: Researchers used two rounds of qualitative coding, reducing an initial 195-code codebook to 62 codes grouped into six themes.Two researchers developed and reconciled the codebook through repeated discussions, without calculating inter-rater reliability because the goal was to identify emergent themes.
- Methodological limitations: The interview sample may underrepresent highly sensitive disclosures and is subject to recall and social desirability biases.Some participants may have avoided discussing especially sensitive conversations, and reported experiences outside prepared conversations may be ambiguous.
5 INTERVIEW RESULTS
Interview participants disclosed diverse personal and interpersonal information while weighing task utility, convenience, and privacy concerns. They navigated these tensions by accepting risks, avoiding sensitive tasks, or adopting ad-hoc protective measures.
- Participants shared PII, health, financial, legal, emotional, and other people’s data across varied conversational tasks.Examples included zip codes for relocation advice and physiological data for tailored diet and exercise plans.
- Perceived capability, convenience, and emotional support encouraged disclosure, whereas doubts about competence or cumbersome input discouraged it.Participants sometimes treated ChatGPT as a friend or therapist, while operational barriers led users to submit partial documents.
- Disclosure decisions also depended on perceived identifiability and personal sensitivity, which varied substantially across participants and tasks.Some users withheld phone numbers, addresses, or birth dates, while others shared names or altered data such as weight.
- Participants cited institutional misuse, social discovery, idea theft, third-party access, and workplace policies or NDAs as privacy concerns.Concerns included marketing, human review, public disclosure, redistribution of work, and sharing confidential company information.
- Users prioritized task objectives differently: some accepted privacy risks for benefits, while others avoided sensitive tasks and received less helpful results.Nearly everyone also used one or more ad-hoc privacy-protective measures to seek a middle ground between privacy and utility.
6 DISCUSSION
The discussion synthesizes user-driven privacy threats in LLM-based conversational agents and argues that privacy protection is constrained by system design, user mental models, and structural incentives.
- User-Driven Privacy Threats: Users raised traditional institutional privacy concerns, while fewer participants reported LLM-specific memorization concerns.Traditional concerns included incomplete deletion and human review of user data.
- Design and Privacy Controls: ChatGPT’s opt-out design bundled training and chat-history controls, while the alternative form was difficult to discover and unavailable to participants.The settings option was found by all but P15; none found the FAQ-linked alternative.
- User-Driven Privacy Threats: LLM-based conversational agents enable unprecedented sharing of other people’s information, creating severe interdependent privacy concerns.Novel use cases include drafting responses to colleagues’ emails.
- User-Driven Privacy Threats: Users’ privacy judgments were contextual and subjective, but one end-to-end model served requests across differing sensitivity contexts.The discussion connects this one-size-fits-all design to challenges in supporting context-sensitive privacy decisions.
- Human-Like Interactions: Human-like interactions can encourage gradually increasing disclosure of sensitive information and nudge what users share.The finding extends prior work on social relationships with non-human agents.
- Design and Privacy Controls: Users applied privacy-protective measures such as omitting, obfuscating, lying, and segmenting inputs, motivating automated or semiautomated assistance.Manual desensitization can be tiring and users may forget to perform it.
- Prospects for Privacy: Participants held serious misconceptions about LLM operation and data use, making transparency for end users an unresolved prerequisite for ethical deployment.The discussion identifies mass education about LLM-based systems as a significant challenge.
- Prospects for Privacy: The authors caution that incremental privacy-control improvements may have limited broad effectiveness because cognitive biases, opacity, binary choices, and social interaction impede privacy self-management.They also question whether privacy-preserving guidelines will be meaningfully adopted under data-collection incentives.
7 CONCLUSION
The conclusion presents a human-centered study of LLM privacy concerns, combining real-world ChatGPT disclosures with interviews to characterize users’ trade-offs and design challenges.
- Approach: The study analyzed ShareGPT52K conversations and interviewed 19 LLM-based conversational-agent users about privacy concerns and disclosure behaviors.The interviews contextualized participants’ behaviors using their real-world chat logs.
- Findings: Users wanted to protect privacy when possible but contended with perceived trade-offs involving convenience and utility.The one-size-fits-all model placed much of the responsibility for privacy protection on users.
- Implications: Flawed mental models and dark patterns in ChatGPT’s opt-out interface made user privacy protection difficult.The conclusion identifies these as central problems requiring attention from HCI and LLM research communities.
A.1 Rules for Determining Non PII vs. PII
The annotation rules distinguish personally identifiable information from non-PII using data validity, public availability, identifiability, and inferential risk.
- Non-PII: Invalid data types and public information not linked to a specific individual are labeled non-PII.Examples include public figures and information available online.
- PII: Ambiguous names are generalized to realistic cases and labeled PII when the context indicates a real person rather than a placeholder.“John Doe” is treated as a likely placeholder example.
- Privacy Assessment: Information is treated as more private when access is restricted to direct contacts or it is linked to an identifiable person.Online availability alone does not make information non-PII if it remains directly linked to an individual.
A.2 Rules for Determining Actor Type (Self/Others/Both/Unknown)
The actor-type rules classify PII according to whether it belongs to the user, other people, both, or an uncertain subject.
- Contextual Evidence: When the relationship between the user and mentioned people cannot be determined, the case is labeled Unknown rather than assigned an actor type.Clear contextual evidence is required for Self, Others, or Both.
- Actor Labels: Self identifies PII belonging to the ChatGPT user, while Others identifies PII belonging to other people.The classification follows contextual evidence about ownership.
- Actor Labels: Both applies when PII concerns the user and third parties, while Unknown applies when ownership is uncertain or information is limited.Multiple possible categories also trigger Unknown pending further clarification.
- Contextual Evidence: Explicit ownership phrases such as “my information” support Self labeling, while task context can imply that a URL belongs to the user.The rule permits contextual inference when the conversational task is highly specific.
A.3 Rules About PIIs not Detected by the Algorithm
Undetected PII should still be labeled when reviewers identify it during analysis.
- Reviewers should label PII that the detection algorithm misses.
A.4 Rules about Specific Categories of PII
The codebook specifies category-specific rules for identifying and labeling PII, with contextual judgment required for ambiguous cases.
- First names, last names, full names, and aliases all count as PERSON PII.
- Current dates in web-search results are labeled Self, interpersonal timestamps Both, and transaction histories Self.
- Private IP addresses are excluded from the PII category.
- URLs identifying a person, such as a company website, are PII, while tracking-parameter URLs are labeled Unknown.
- NRP and LOCATION require contextual judgment about whether they identify someone in a relatively private setting.
B CODEBOOK FOR DATASET ANALYSIS
The dataset-analysis codebook organizes ChatGPT use cases, disclosure patterns, privacy risks, and the interview procedures used to study them.
- Use cases and disclosure patterns: The codebook covers direct questions, interactive task definition, and detailed background disclosure as distinct interaction patterns.
- Use cases and disclosure patterns: Interactive conversations can produce progressively greater disclosure as users provide more information to complete complex tasks.
- Use cases and disclosure patterns: Providing extensive background context increases data sharing and may enable inferences about interests, occupation, or location.
- Interview protocol: The interview protocol elicited users’ goals, covered information, privacy concerns, protective strategies, and difficulties encountered.
- Mental-model exercise: Participants also completed a drawing exercise to explain how ChatGPT processes submitted data and whether past input could influence later responses.
D CODEBOOK FOR THE INTERVIEW RESULTS
The interview codebook categorizes users’ disclosure contexts, privacy concerns, trade-off strategies, mental models, and reactions to memorization and privacy controls.
- Disclosure norms and influences: Disclosure decisions depend on perceived CA capability and operational convenience, while work-related data is constrained by company policies or NDAs.
- Perceived risks and norms: Users report concerns about others discovering their AI use, stealing ideas, accessing others’ data, and exposing their own data.
- Disclosure trade-offs: Users balance disclosure benefits against privacy risks by accepting risks, avoiding tasks, sanitizing inputs, or seeking only general advice.
- Mental models: Mental models range from viewing ChatGPT as magic or a searcher to understanding it as an end-to-end model, with differing beliefs about training data.
- Memorization risks: Users reported memorization experiences and concerns, but some dismissed risks because they shared no sensitive data, could not imagine memorization, or expected inaccurate outputs.
- Privacy-protective support: Users lacked awareness of available privacy-protective options, encountered dark patterns around opt-out, and wanted more granular controls.