Source-linked AI summary
What Does it Mean for a Language Model to Preserve Privacy?
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, Florian Tramèr
TL;DR
Language models process natural-language data that can encode private information, yet they may memorize and expose training examples through inference and extraction attacks. The paper analyzes data sanitization and differential privacy against the contextual and distributed nature of language privacy, arguing that neither provides a generic meaningful guarantee. It concludes that privacy-preserving models should rely on text explicitly intended for public use.
Problem
Language models can memorize and expose private training information, while natural-language privacy depends on context, unclear information boundaries, and shared authorship.
Method
The paper critically examines the assumptions and privacy claims of data sanitization and differential privacy for language data.
Results
Existing methods provide limited data protection for specific text types but cannot fully satisfy human privacy expectations for natural language.
Takeaways & Limitations
The paper concludes that language models should be trained exclusively on text explicitly intended for fully public use.
Takeaways & Limitations
User-level differential privacy cannot erase many private facts because they may be distributed across text written by multiple users.
Abstract
from arXiv · showhide
Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets. An adversary can exploit this tendency to extract training data. Depending on the nature of the content and the context in which this data was collected, this could violate expectations of privacy. Thus there is a growing interest in techniques for training language models that preserve privacy. In this paper, we discuss the mismatch between the narrow assumptions made by popular data protection techniques (data sanitization and differential privacy), and the broadness of natural language and of privacy as a social norm. We argue that existing protection methods cannot guarantee a generic and meaningful notion of privacy for language models. We conclude that language models should be trained on text data which was explicitly produced for public use.
1 Introduction
Natural language carries context-sensitive personal information, while language models can memorize and reproduce training text without recognizing when disclosure violates privacy. The paper argues that sanitization and differential privacy rely on assumptions that do not fit natural language, so meaningful privacy requires training on text explicitly intended for public use.
- Privacy and language: Natural-language privacy depends on context, because sharing the same information with one audience may violate privacy in another.Human communication relies on implicit boundaries about appropriate recipients, timing, and use.
- Privacy and language: Language models create far-reaching privacy risks by memorizing and regurgitating training data, especially atypical data points.Their growing size and use of large natural-language datasets expand the scope of potential exposure.
- Existing protections: Existing approaches either remove presumed private information through data sanitization or reduce memorization using differential privacy.Neither approach is designed to make models understand the contextual appropriateness of sharing.
- Existing protections: Sanitization and differential privacy assume private information and protected records can be formally specified, recognized, and mapped to clear boundaries.These assumptions often fail for natural-language data, where sensitive information may be distributed across text and contexts.
- Conclusion: The paper argues that current methods offer limited data protection but cannot fully satisfy human privacy expectations for shared text.It proposes contextual integrity as the target and concludes that models should use data explicitly intended for fully public use.
- Conclusion: Public availability does not necessarily mean language data was intended for broad public use, so training decisions require understanding context.Public posts, leaked material, and data shared to harm others may still carry privacy expectations.
2 Background on Language Models
Language models learn statistical relationships among tokens from large text corpora and support many natural-language applications. Their rapid scaling in model size, data, and compute improves performance but increases the amount of potentially private language processed.
- Language-model foundations: Language models learn token occurrence probabilities and word-sequence relationships from large text corpora.They extract statistics and correlations at sentence and paragraph levels for word-sequence prediction.
- Applications: Language models support applications including call centers, medical systems, assistants, autocomplete, and document processing.These applications operate in settings where personal and sensitive language data is created and used.
- Language-model foundations: A token is a character sequence grouped as a useful semantic processing unit, such as a character, word, or sub-word.The paper uses tokens as the basic units whose occurrence relationships language models learn.
- Scaling trends: Figure 1 links increasing model and training-data sizes with improved performance and growing privacy risks.The caption reports that top-model size increases by an order of magnitude each year and that large training sets are required.
- Language-model foundations: Language models construct plausible sentences by learning local word roles and global consistency across longer sequences.Embedding functions additionally map words and phrases to vectors whose closeness reflects semantic similarity.
3 Privacy Risks of Language Models for their Training Data
Language models can memorize training examples, enabling attacks that infer membership or extract sensitive text. The paper presents these risks as motivating privacy-preserving training, while arguing that current sanitization and differential-privacy approaches are inadequate for language data.
- Memorization risks: Memorization can leak private training information because model behavior differs between examples seen during training and unseen examples.Memorization may also support generalization to long-tailed distributions, creating a tension between utility and privacy.
- Inference attacks: Membership-inference attacks determine whether a data point was used to train a model by querying its behavior.They can identify atypical training members, which are more likely to contain sensitive information.
- Training-data extraction: The largest GPT-2 variant yielded a successful extraction of a person’s full name, address, and phone number from training data.In BERT models trained on de-identified clinical notes, more than 4% of generated sentences containing a patient’s name also contained a true medical condition.
- Implications: Inference attacks are improving, so current attack results may underestimate the privacy risks of large language models.This motivates training methods that learn language structure without memorizing sensitive information.
- Protection approaches: Existing privacy-preserving methods use either data sanitization to remove private text or differential privacy to mitigate memorization.The paper argues that neither class adequately preserves users’ privacy in language models.
4 What does preserving privacy in language modeling require?
Preserving privacy in language modeling requires judging whether information is appropriate to reveal to particular people in particular contexts, rather than simply avoiding memorization. This is difficult because privacy depends on context, meaning, repetition, evolving norms, ownership, and group boundaries that language models do not reliably understand.
- Privacy-preserving language models should reveal secrets only in appropriate contexts and to appropriate people.Determining privacy requires identifying the relevant context, the secret itself, and the people who know it.
- Secrets are contextual: Contextual privacy judgments require information beyond the text, including conversational and sociocultural context and the intended sharing scope.The same information may be acceptable for one audience but a violation when shared more broadly.
- Secrets are hard to identify: Private information is difficult to identify because equivalent meanings can appear in different forms, including synonyms, spelled-out formats, and indirect references.An allusion such as asking whether someone is getting a divorce can preserve the same sensitive meaning while making automatic recognition harder.
- Secrets are hard to identify: Repeated references can remain private, so protecting them requires recognizing the shared sensitive topic across multiple nonduplicate text items.Repeated discussion among a group does not necessarily make information non-sensitive to outsiders.
- Secrets evolve: Privacy boundaries change as language, social norms, and the status of secrets evolve over time.Information may be confidential during an investigation but later public, while source identities may remain secret indefinitely.
- In-groups and ownership: Identifying who belongs to a secret’s in-group is difficult because text can concern multiple people and the permitted audience varies across secrets and contexts.A person may also share someone else’s private information, making ownership and audience boundaries nontrivial.
- Human notions of privacy: Humans preserve privacy through judgments about whether sharing is appropriate for a given party and context, not merely by failing to memorize secrets.Applying this approach to language models would require intrinsic understanding of language and social contexts beyond existing methods.
5 A Critical Analysis of Privacy Technologies for Language Models
The paper argues that data sanitization and differential privacy rely on data boundaries and privacy definitions that do not fit natural language, limiting their ability to preserve meaningful privacy.
- Data sanitization: Data sanitization struggles because private information is context-dependent, non-discrete, and often lacks identifiable borders.It works best for well-formatted information such as social security numbers, but struggles with broad discussions and secrets whose scope depends on context.
- Data sanitization: Sanitization can reduce risk for context-independent, well-defined, static secrets, but cannot claim generic privacy for free-form text.The method remains useful as an obfuscation technique when the information to remove is specified in advance and has clear boundaries.
- Differential privacy: Differential privacy’s guarantee depends on defining records whose addition or removal corresponds to the private information being protected.Word- or sentence-level records are often inadequate for secrets spanning multiple records, while user-level records may still fail to capture shared information.
- Differential privacy: Group-level privacy guarantees weaken exponentially with group size, even though information shared among many people can remain equally sensitive.The paper notes that k users receive kε-DP, while contextual sensitivity does not necessarily diminish as the group grows.
- Differential privacy: Achieving secret-level differential privacy would require understanding the context and boundaries of secrets that differential privacy is designed to avoid specifying.Thus, stronger semantic protection would reintroduce the difficult identification problem underlying language privacy.
6 Summary and Discussion
The paper presents human privacy norms as too nuanced for current language-model safeguards and questions whether public data or individual consent resolves these problems. It proposes public-intended training data with local private personalization as a promising alternative.
- Summary and Discussion: Human privacy norms vary with context, relationships, time, and changing groups, making private information difficult to define for language-model training.Private information can change form and be shared among groups whose membership changes with human relationships.
- Summary and Discussion: Existing privacy-enhancing techniques provide guarantees that are not meaningful enough for language data because their assumptions about private information are too narrow.The paper distinguishes formal data protection from privacy expectations concerning how language should be emitted and used.
- Summary and Discussion: Public accessibility does not imply public intent: web data may be leaked, inadvertently exposed, or shared for a limited audience, while LM memorization can violate its intended context.The paper therefore recommends training on data explicitly intended or allowed for public dissemination.
- Summary and Discussion: Informed consent is difficult because people often cannot understand downstream data handling, and sensitive information may be jointly held by multiple users.Consent would need to account for all people implicated by shared information, not only the person whose data is collected.
- Summary and Discussion: Private personalization combines exclusively public-intended pretraining with local fine-tuning on a user’s private data when the model remains local to that user.The paper presents this as a promising middle ground whose main user privacy risks would be alleviated under local use.
7 Conclusions
The paper calls for rigorous privacy expectations and meaningful guarantees for language data, arguing that existing protection methods and arbitrary consent do not provide reliable privacy assurance. It concludes that truly privacy-preserving language models should rely exclusively on data intended to be public.
- 7 Conclusions: The paper argues that data protection is not equivalent to privacy because existing methods do not reliably align with users’ privacy expectations for language data.It also states that users cannot properly consent to arbitrary computations on their data.
- 7 Conclusions: The paper concludes that language models should rely exclusively on text intended to be public as the only truly privacy-preserving solution it identifies.This conclusion follows the paper’s broader call for rigorous privacy guarantees tailored to language data.