Source-linked AI summary
Are Large Pre-Trained Language Models Leaking Your Personal Information?
Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang
TL;DR
The paper asks whether PLMs leak personal information and studies this question by querying for email addresses using training contexts or prompts containing owners’ names. It finds leakage through memorization, while weak association makes specific-person extraction risk low, although potential privacy risks remain.
Problem
The paper asks whether PLMs are prone to leaking personal information, given evidence that large models may memorize sensitive training data.
Method
The study queries PLMs for email addresses using email contexts to test memorization and owner-name prompts to test association.
Results
PLMs leak personal information through memorization, but specific personal-information extraction risk is low because models cannot meaningfully associate information with its owner.
Takeaways & Limitations
Researchers should distinguish memorization from association when evaluating PLM privacy risks and assess trained models before releasing them.
Takeaways & Limitations
Privacy concerns restrict the data that can be used, and the study focuses on email addresses rather than other personal information.
Abstract
from arXiv · showhide
Are Large Pre-Trained Language Models Leaking Your Personal Information? In this paper, we analyze whether Pre-Trained Language Models (PLMs) are prone to leaking personal information. Specifically, we query PLMs for email addresses with contexts of the email address or prompts containing the owner's name. We find that PLMs do leak personal information due to memorization. However, since the models are weak at association, the risk of specific personal information being extracted by attackers is low. We hope this work could help the community to better understand the privacy risk of PLMs and bring new insights to make PLMs safe.
1 Introduction
PLMs raise privacy concerns because they may memorize personal information, but it remains unclear whether attackers can extract information tied to specific individuals. This paper distinguishes memorization from association and finds leakage through memorization while specific-person extraction risk is low.
- PLMs can expose names, email addresses, and phone numbers from training data, creating privacy risks for private and even public corpora.
- Prior work shows that memorized information may be difficult to extract effectively when models cannot associate it with the relevant owner.
- The paper distinguishes memorization, recovering information with a specific prefix, from association, querying information using the owner’s name.
- The study focuses on email addresses because they are important communication identifiers linked to fraud and spam risks.
- PLMs leak personal information in some situations through memorization, but weak association makes extracting a specific person’s information difficult.
2 Related Work
Related work establishes that PLMs can retrieve knowledge and memorize training data, creating potential privacy risks. Existing approaches seek to reduce these risks through privacy-preserving training or sensitive-information removal.
- Prior studies show that large PLMs contain substantial knowledge recoverable through appropriately designed prompts.
- Research finds that PLMs memorize large portions of training data, allowing sensitive information to leak in some situations.
- Privacy-preserving approaches include differential-privacy guarantees during training and removing sensitive information from training corpora.
3 Problem Statement
The paper formalizes personal-information leakage through two distinct model capacities: memorization and association. It measures them by prompting models with training contexts or attacker-designed prompts containing an owner’s name.
- The task is to measure the risk that PLMs leak personal information through memorization and association.
- Memorization means that a training-data sequence can prompt the model to produce personal information using greedy decoding.
- Association means that an attacker-designed prompt, usually containing the owner’s name, can prompt the model to produce personal information using greedy decoding.
- The experiments quantify memorization with target contexts and measure association by querying models with varied attacker prompts.
4 Data and Pre-Trained Model
The study evaluates GPT-Neo models on email information derived from the Enron Corpus. Email addresses are extracted from messages and mapped to owners’ names for testing.
- The study tests GPT-Neo models with 125 million, 1.3 billion, and 2.7 billion parameters, pretrained on the Pile.
- The Pile includes the Enron Corpus among 22 diverse high-quality datasets used for pretraining.
- The Enron Corpus contains over 600,000 emails generated by Enron employees.
- The corpus is processed to extract email addresses from message bodies and map them to owners’ names using the UC Berkeley Enron Database.
5 Method
The experiments measure personal-information leakage through memorization and association by prompting GPT-Neo with email contexts, names, and demonstrations. Outputs are generated with greedy decoding, and email addresses are extracted for evaluation.
- For GPT-Neo, the procedure generates 100 tokens, extracts the first matching email address with regular expressions, and uses greedy decoding by default.
- The study measures memorization by prompting models with 50, 100, or 200 tokens preceding a target email address in the training corpus.
- Association is evaluated mainly in a zero-shot setting using manually designed prompts that contain an owner’s name and request an email address.
- The prompts include direct name-to-email queries, name-and-email fields, mailto formats, and original-message formats.
- A domain-informed zero-shot prompt supplies a known email domain while asking the model to complete the target person’s address.
- K-shot experiments provide true name–email pairs as demonstrations, with examples sampled either randomly or from the target’s domain.
6 Result & Analysis
The experiments show that PLMs can recover memorized email addresses, but their ability to associate names with addresses is weak. Attack success increases with longer contexts, known domains or examples, and larger models, while practical risk remains constrained in several settings.
- Context setting: 8.80% of email addresses were predicted correctly by GPT-Neo in the context setting, including addresses that did not follow standard patterns.The context setting uses prefixes from training data, making these predictions meaningful for measuring memorization.
- Memorization and association: 0-shot (D) outperformed 0-shot (C) because its longer prefix exposed more memorized sequence information.The two prompts differ in prefix length, supporting memorization rather than association as the explanation for the performance gap.
- Memorization and association: 0-shot (D)-[2.7B] achieved 0.19% accuracy on email addresses absent from the training corpus, compared with 1.24% in Table 2.The large decrease supports the role of memorization or association in recovering addresses from the original setting.
- Knowledge and scale: Known domains or additional examples substantially improved performance, while larger models usually improved accuracy across settings.More examples reinforce email-format patterns, and scaling may increase memorization and prompt understanding.
- Unknown-domain setting: PLMs predicted only a very small number of email addresses correctly when the domain was unknown, with most correct predictions matching identified patterns.The results indicate that many predictions can be explained by memorized or guessable address formats rather than exact name-address associations.
- Practical risk: Known-domain predictions were no better than a simple rule-based method, and many correct addresses followed patterns attackers could guess directly.For context-based predictions, attackers either lack access to private contexts or can retrieve the address after finding the context without using PLMs.
7 Mitigating Privacy Leakage
The paper proposes mitigating PLM privacy leakage before, during, and after training, while information owners can reduce exposure through safer email practices.
- Training PLMs: Pre-process training data by removing or blurring risky long patterns and deduplicating data to reduce memorized text.The paper identifies long extraction-helpful patterns as risks and cites deduplication as a way to substantially reduce memorization.
- Training PLMs: Train PLMs with differentially private stochastic gradient descent to mitigate privacy risks during model training.
- Information owners: Information owners should avoid publishing personal information directly in text form and should avoid email addresses with obvious patterns.Suggested alternatives include using an image or rewriting the address with recovery instructions.
8 Conclusion
The paper distinguishes memorization from association as separate sources of privacy leakage in PLMs. It finds leakage through memorization, but low risk of specific information being linked meaningfully to its owner, and proposes defenses.
- The paper presents the first distinction between memorization and association in pre-trained language models.
- PLMs leak personal information through memorization, but specific personal information has low leakage risk because models cannot meaningfully associate it with the owner.
- The paper suggests defense techniques to mitigate potential threats and improve understanding of PLM privacy risks.
Limitations
The study’s privacy-sensitive design constrains the data it can use, while the authors believe its methods and findings may generalize beyond email addresses and private data.
- Privacy concerns impose limitations on the research, including constraints on the data that can be used.
- The study chooses email addresses because they represent personal information, have a fixed format for extraction and analysis, and are available in the Enron Email Dataset.
- The authors believe the methods and findings can generalize to other personal information and private data because models are trained similarly.
Ethics Statement
The paper frames personal-information leakage as an ethical privacy concern and uses public data and models, with anonymized email addresses, to reduce potential harms.
- Leaked personal information such as email addresses can cause privacy issues including email fraud and spam.
- The analysis uses data and models that are already public to minimize ethical concerns and support reproducibility.
- The paper replaces real email addresses with consecutive characters such as abcd to protect privacy.
- Despite low risk of specific information leakage through weak association, the authors caution against underestimating memorization and potential association threats.
A Rule-Based Method
The paper uses a rule-based baseline built from email-address patterns, then compares decoding algorithms and examines whether frequency relates to correct predictions.
- Pattern construction: 28 email-address patterns are classified by the length of the owner’s name, with Z denoting addresses that fit none of them.The rule-based baseline is designed for settings where the target domain is known.
- Pattern construction: The rule-based method predicts the most frequent compatible pattern, or uses the zero-shot prediction when no pattern matches the target name.
- Decoding algorithms: Top-k sampling performs worse than greedy decoding, while beam search and greedy decoding have similar performance.The reported settings use top-k sampling with k = 50 and temperature = 0.7, and beam search with num_beams = 5 and early stopping.
- Frequency effects: Correctly predicted email addresses have higher mean and median frequencies than the full dataset, indicating that frequent addresses are more likely to be memorized and associated.Statistics are reported for GPT-Neo 2.7B, excluding settings with fewer than 20 correct predictions.