PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild

Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, Golnoosh Farnadi

arXiv:2407.11438v2cs.CL

TL;DR

Users have limited transparent controls over how chatbot conversations are used or shared, and evidence about sensitive disclosures in these interactions remains limited. The paper analyzes naturally occurring conversations, builds taxonomies of sensitive information and tasks, and finds frequent PII and non-PII disclosures, including information PII detectors do not capture.

  • Problem

    Commercial chatbots lack transparent controls for tracking conversation use and sharing, while prior studies had limited and biased access to user data and left disclosures poorly understood.

    Source: §1 · pp. 1–2

  • Method

    The authors analyze naturally occurring WildChat conversations and create task and sensitive-topic taxonomies, with automatic annotations for 5k conversations validated against a subset of human annotations.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Over 70% of queries contained detected PII and almost 15% mentioned non-PII sensitive topics, while PII detection misses other sensitive disclosures such as sexual content and drug use.

    Source: §1 · p. 2, §3.3 · p. 6

  • Takeaways & Limitations

    The authors release taxonomies and annotations to support future privacy research and call for greater attention to risks in human-chatbot conversations.

    Source: §7 · p. 11

    The authors note that WildChat may overrepresent sensitive or disallowed activity because some users may use it to mask their activity.

    Source: §9 · p. 12

Abstract

from arXiv · show

Measuring personal disclosures made in human-chatbot interactions can provide a better understanding of users' AI literacy and facilitate privacy research for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users to commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) PII detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such as detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.

1 Introduction

The introduction frames user-chatbot disclosure as a privacy concern amid limited transparency and evidence, then presents an analysis of sensitive information and the contexts in which it appears. It reports frequent detected PII and non-PII sensitive disclosures, including in translation tasks.

1 Introduction

  • Commercial LLM chatbots serve both work and personal tasks, but users and researchers lack transparent controls for tracking how conversations are used or shared.The introduction links this opacity to concerns about users losing control over how their data is shared, regulated, and passed on.Source: §1 · p. 1
  • Collected conversations may be used for model updates or other company purposes, and can be exposed through memorization, data breaches, or ransomware attacks.The authors call for nudging mechanisms to help users moderate interactions and for more research on local, private models.Source: §1 · pp. 1–3
  • The analysis asks what sensitive information users share, how frequently it appears and can be detected, and which tasks provide its context.It also explores limitations of PII detection and sensitive information outside traditional PII categories.Source: §1 · p. 2
  • The study develops taxonomies of sensitive information and disclosure contexts because prior work had limited, biased access to user data and left PII disclosure poorly understood.The authors describe automatic categorizations for 5k WildChat conversations, validated against a subset of human annotations, and release them for future research.Source: §1 · p. 2
  • Over 70% of queries contained detected PII and almost 15% mentioned a non-PII sensitive topic, while around 50% of translation queries contained detected PII.These rates were observed even though WildChat had undergone one round of PII removal.Source: §1 · p. 2

2 Data and Methods

The study uses naturally occurring WildChat conversations, with ShareGPT as a comparison, to examine the tasks in which users disclose information. It builds task categories through iterative annotation, applies GPT-4 to filtered samples, and compares task distributions across datasets and sampling choices.

2.1 Data

  • The study compares naturally occurring human-GPT conversations from WildChat with conversations users opted to share from ShareGPT.WildChat contains one million conversations; the study filters out non-English conversations because its methods rely on English-trained tools, and notes that the dataset may incentivize sensitive or disallowed tasks.Source: §2.1 · p. 3, §2 · p. 3

2.2 Task Annotation

  • The authors iteratively hand-annotated 300 conversations, then settled on 21 task categories to represent users' goals.They used GPT-4 to categorize 5,000 filtered WildChat conversations and evaluated predictions by manually checking 20 predicted examples per category; mean accuracy was 89.2%, and three categories scoring below 70% were excluded.Source: §2.2 · pp. 3–4

2.3 Task Distribution

  • Explanation is common in WildChat, while story generation becomes the most frequent task without IP-address filtering, indicating concentration among power users.ShareGPT contains more explanation, information retrieval, and code generation than WildChat, likely reflecting users' selection of conversations to share.Source: §2.3 · pp. 4–5, Figure 2 · p. 4

3 How much detectable PII do users share?

The section measures detectable PII with Azure AI Language annotations and examines how those detections vary across datasets and tasks. It finds that detected PII is common, but detector errors and sensitive topics outside traditional PII categories limit what PII detection alone captures.

3.1 PII Detection

  • Azure AI Language annotates more than 20 fine-grained PII categories in both datasets, and the authors manually check for errors and drop erroneous categories.The service is used to detect PII in selected subsamples of WildChat and ShareGPT.Source: §3.1 · p. 5

3.2 Detected PII Distribution

  • Across WildChat and ShareGPT, most queries contain detected PII, with names and organization names accounting for the largest share.WildChat had already undergone one PII-removal round, but the authors report that the rule-based Presidio system often misses poorly formatted PII.Source: §3.2 · p. 5, Figure 3 · p. 6
  • The authors dropped several Azure AI categories, including national ID, passport numbers, and SWIFT codes, because of high error rates.They retained the IBAN category even though none of its labels were international banking numbers; the labeled spans were PII, mostly API or subscription tokens.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3 Is PII detection sufficient for privacy?

  • PII detection misses sensitive disclosures outside traditional PII categories, including sexual preferences, verbatim private messages, and personal habits or drug use.Examples include explicit sexual content in story and script generation, work- and finance-related messages in communication generation, and personal habits or drug use under explanation and how-to tasks.Source: §3.3 · p. 6

4 Sensitive Topic Detection

The section develops sensitive-topic categories to assess disclosures that traditional PII detection may miss. It examines which topics escape detection and the conversational tasks in which sensitive topics appear.

4.1 Discovering sensitive topics

  • The taxonomy covers topics including academic information, fandoms, job and visa applications, and erotic content.Table 2 presents the full taxonomy with example queries assigned labels by GPT-4.Source: §4.1 · p. 7, Table 2 · p. 7
  • GPT-4 predictions were evaluated on the same 5,000 conversations, with a mean accuracy of 87% across retained topics.The authors hand-annotated 20 random positive predictions per topic and discarded quoted emails and messages for accuracy below 70%.Source: §4.1 · p. 7

4.2 Where does PII detection fall short?

  • PII detectors flag many names in storytelling and erotic topics, yet detect at best a minority of sensitive topics such as fandom and hobbies.Names in these contexts may be fictional or sensitive, while topics such as fandom and hobbies may not be captured by PII detection.Source: §4.2 · pp. 7–8

4.3 In what conversational contexts are sensitive topics mentioned?

  • Jailbreaking, role-playing, and story-generation tasks frequently contain erotic content, while communication tasks more often involve financial or corporate information, applications, and relationships.Fandom mentions are frequent in role-playing, story-generation, and song or poem generation.Source: §4.3 · p. 8, Figure 7 · p. 9

5 Discussion

The discussion connects privacy risks to storytelling, data practices, and users’ relationships with chatbots. It recommends privacy safeguards and nudges that can respond to the task and context.

Design implications

  • Designers should protect data across deployment stages and consider local models, encrypted data, and privacy-preserving methods.The discussion recommends anonymized, safely stored data and suggests local disclosure detection to warn users before data is sent to the cloud.Source: §5 · p. 9
  • Task-responsive nudges could highlight detected PII or warn about particular tasks, alongside clear choices about sharing and storing conversations.The discussion also recommends informing users about data collection and potential uses such as model training.Source: §5 · p. 9

Relationship to self-disclosure

  • WildChat data cannot reveal whether users view the chatbot, model, company, or researchers as their interlocutor.The authors call for more HCI research to disentangle users’ perceptions of relationships and trust with chatbots.Source: §5 · p. 10

Sexually explicit storytelling

  • Story prompts often contain names, ages, and locations that detectors may flag, while erotic topics can be sensitive even when personal details are absent.The authors note that stories may involve details about real people known to users and call for greater attention to this frequent use case.Source: §5 · p. 10

6 Related Work

The related work positions this study at the intersection of chatbot privacy research and self-disclosure detection. It distinguishes the study’s focus on implicit sensitive topics and closed user-chatbot conversations from much prior work.

User-chatbot interactions

  • Unlike ShareGPT work that lacked user consent, this study uses WildChat, which collects a wide variety of user interactions with consent.The cited ShareGPT analysis examined conversational-agent usage, including design and planning tasks.Source: §6 · p. 10

Privacy risks with humans and LLMs

  • Prior privacy research examines training-data leakage and mitigation, while this work measures sensitive topics and PII disclosures in chatbot interactions.The authors describe their task taxonomy as complementing prior findings about why people talk to chat assistants.Source: §6 · pp. 10–11

Self-disclosure detection

  • Much self-disclosure detection work targets explicit statements in public online conversations, whereas this study examines implicit sensitive topics in closed chatbot conversations.Prior methods include topic modeling, fine-tuning, multi-task models, and LLM-based prompts.Source: §6 · p. 11

7 Conclusion

The conclusion presents the study as an analysis of PII and sensitive-topic disclosures in chatbot conversations, with released taxonomies and annotations.

7 Conclusion

  • The authors release task and sensitive-topic taxonomies with automatic annotations for a WildChat sample and call for further privacy research.They also report notifying the WildChat authors of their findings.Source: §7 · p. 11

8 Ethical Considerations

The study treats WildChat conversations as deeply personal data with implications for user safety and data security. It documents the dataset’s collection and sharing terms and reports notifying its maintainers about sensitive examples.

8 Ethical Considerations

  • WildChat contains deeply personal self-disclosures, raising data-security questions for researchers, practitioners, and LLM users.The authors hope the results can support safety guidelines, AI literacy, and further research.Source: §8 · p. 11
  • WildChat users consented before interacting and received a use-and-sharing agreement; hashed IP addresses and country locations were publicly released.Users received free model access in exchange for signing the agreement.Source: §8 · p. 11
  • The real-world conversations offer a way to examine disclosure and self-disclosure, and to quantify sensitive information shared with assistants and data-collection risks.Source: §8 · p. 11
  • Before publication, the authors notified WildChat’s maintainers about the sensitive examples they identified.Source: §8 · p. 12

9 Limitations

The authors identify limits related to changing user behavior, English-only focus, possible WildChat selection bias, and privacy risks in concentrated, interconnected downstream applications. These boundaries constrain how broadly findings can be applied and point to areas for future research.

9 Limitations

  • User behavior and interactions with ChatGPT and other models may change over time.Source: §9 · p. 12
  • The findings focus on English speakers, and the authors caution that they may not generalize because current LLM abilities differ across languages.Source: §9 · p. 12
  • Greater trust and more advice-seeking applications may heighten privacy concerns, while model concentration could raise leakage, attack, and breach risks in downstream tasks.The authors call for research on privacy risks from interconnected applications depending on a single LLM.Source: §9 · p. 12
  • WildChat may overrepresent sensitive or disallowed activity if users choose it to mask their activity; the authors accept this limitation given limited access to most conversations.Source: §9 · p. 12

A.1 Preliminaries

The appendix defines key concepts and describes how human annotation and GPT-4 prompts support task and sensitive-topic categorization. It also identifies figures and a table covering geographic relationships and task categorization.

A.1 Preliminaries

  • PII is described as identity-vulnerable information spanning birth-, society-, social-, character-based, and structured categories; LLMs are transformer-based language-modeling architectures used across tasks.Source: §A.1 · p. 20

A.2 Topic Model for Human Annotation

  • For human task annotations, the authors sampled 10 high-probability documents from each of 30 LDA topics trained on 10,000 conversations; Table 3 presents the topics and annotator-assigned tasks.Source: §A.2 · p. 20, Table 3 · p. 21

A.3 GPT-4 Task Prompt

  • The GPT-4 task prompt asks which tasks users explicitly request and requires confidence, reasoning, and evidence, using a supplied task list.Source: §A.3 · pp. 21–22

A.4 GPT-4 Sensitive Topic Prompt

  • The GPT-4 sensitive-topic prompt asks whether messages reveal listed categories, whether disclosures are explicit or implicit, who is involved, and the model’s confidence and reasoning.Source: §A.4 · pp. 22–23

A.5 PII by Geographic Location

  • Figure 8 depicts relationships among sensitive topics, PII, and countries in WildChat.Source: Figure 8 · p. 24

A.6 Full Task Descriptions

Related papers

Loading 2407.11438v2…