Source-linked AI summary
ProPILE: Probing Privacy Leakage in Large Language Models
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, Seong Joon Oh
TL;DR
LLM training on web-scale data raises concerns that sensitive PII may leak, while evidence and safeguards for leakage under specific prompts remain limited. ProPILE probes this risk using data subjects’ PII in black-box settings and tuned prompts with model access, finding that crafted prompts can disclose PII and that white-box tuning substantially increases reconstruction rates. The tool is intended to improve awareness and assessment of PII leakage, although its heuristic evaluation data may contain noise.
Problem
LLM training data may contain PII, but the probability and mechanisms of leakage under specific prompt conditions remain insufficiently understood.
Method
ProPILE uses black-box prompts built from known PII and white-box soft prompts tuned with model access and training data to probe leakage.
Results
ProPILE experiments found diverse training-data PII could be disclosed, while soft prompting increased exact-match reconstruction from 0.0047% to 1.3% using 128 data points.
Takeaways & Limitations
ProPILE supports awareness and assessment of potential PII leakage for data subjects and LLM service providers.
Takeaways & Limitations
The heuristic evaluation-data collection may introduce incorrectly associated data or noise, creating uncertainty in interpreting results.
Abstract
from arXiv · showhide
The rapid advancement and widespread use of large language models (LLMs) have raised significant concerns regarding the potential leakage of personally identifiable information (PII). These models are often trained on vast quantities of web-collected data, which may inadvertently include sensitive personal data. This paper presents ProPILE, a novel probing tool designed to empower data subjects, or the owners of the PII, with awareness of potential PII leakage in LLM-based services. ProPILE lets data subjects formulate prompts based on their own PII to evaluate the level of privacy intrusion in LLMs. We demonstrate its application on the OPT-1.3B model trained on the publicly available Pile dataset. We show how hypothetical data subjects may assess the likelihood of their PII being included in the Pile dataset being revealed. ProPILE can also be leveraged by LLM service providers to effectively evaluate their own levels of PII leakage with more powerful prompts specifically tuned for their in-house models. This tool represents a pioneering step towards empowering the data subjects for their awareness and control over their own data on the web.
1 Introduction
Web-scale LLM training data may contain sensitive PII, extending privacy concerns beyond users who knowingly shared information. ProPILE lets data subjects and service providers probe potential PII leakage, and experiments show strategically refined prompts can disclose PII.
- Web-crawled training data may include sensitive PII from personal pages, social media, forums, and online databases.
- LLM services broaden the affected population to virtually anyone whose PII appears on the web, without assurance that adequate safeguards prevent disclosure.
- ProPILE uses a data subject’s known PII to formulate prompts that assess possible inclusion and leakage in deployed LLM products.
- Service providers can use ProPILE with prompts tuned to in-house models to assess leakage and address potential privacy vulnerabilities.
- Experiments on OPT trained on the Pile found that strategically crafted prompts can disclose a significant portion of diverse PII in training data.
- Prompt refinement, model-parameter access, and a few hundred training points can significantly magnify the degree of PII leakage.
2 Related Works
Related work established that learned models can reconstruct sensitive training information and support membership inference, while LLMs intensify these concerns through web-scale data collection. Prompt engineering and soft prompt tuning provide foundations for ProPILE’s white-box probing approach.
- Prior research studied training-data reconstruction and model inversion, including sensitive-attribute reconstruction with black-box access.
- Membership inference offers an evaluation-friendly task for testing whether samples were included in training data across text- and image-generation models.
- Billion-scale LLMs heighten privacy concerns because their web-crawled training data can include personal pages, social accounts, profiles, and in-house email databases.
- Prompt engineering improves downstream performance without fine-tuning, while soft prompt tuning learns token embeddings with the LLM frozen.
- ProPILE’s white-box approach uses soft prompt tuning to refine hand-crafted prompts from its black-box approach.
3 ProPILE: Probing PII Leakage of Large Language Models
ProPILE profiles linkable and structurally different PII leakage through black-box and white-box probing strategies, then quantifies disclosure using generated strings or likelihoods.
- PII attributes: ProPILE evaluates whether a target PII item can be reconstructed from the data subject’s remaining PII items.Linkable leakage compares reconstruction likelihood conditioned on the other PII items with the unconditional likelihood of the target item.
- PII attributes: Structured PII follows recognizable patterns, including phone numbers, email addresses, and physical addresses.These patterns may support regular-expression detection, purging, output detection, and redaction, although complete removal remains difficult.
- PII attributes: Unstructured PII lacks an easy regular-expression pattern and includes family relationships and affiliation information.The paper identifies these cases as posing greater threats of unrecognized privacy leakage than structured PII.
- Probing strategies: Black-box probing creates K prompt templates from the remaining PII, sends them up to N times, and receives sampled responses with likelihood scores when available.The actor has only prompt-and-response access and seeks evidence that one of their own PII items may leak.
- Probing strategies: White-box probing uses soft prompt tuning with training data, model parameters, and gradients to induce more leakage than handcrafted black-box prompts.The learned soft embedding is optimized for expected reconstruction likelihood over a subset of training data, then prepended to prompts containing unseen subjects’ PII.
- Leakage quantification: Leakage is quantified either by exact string match between generated text and target PII or by reconstruction likelihood conditioned on the query prompt.Likelihood scores additionally support estimating the chance of reconstruction under repeated sampling at deployment scale.
4 Probing Existing LLMs
ProPILE probes PII leakage in OPT-1.3B using black-box prompts and white-box soft prompt tuning. Black-box probing reveals linkable PII, while refined prompts, limited tuning data, and model access substantially increase reconstruction.
- Experimental setup: OPT-1.3B was evaluated on structured phone, email, and physical addresses plus unstructured family-relationship and university information extracted from the Pile dataset.Structured PII was collected as linked quadruplets; unstructured PII was extracted with question answering and a confidence threshold above 0.9.
- Black-box probing: Black-box probing compares target-PII likelihood against randomly selected null PII using multiple templates and beam-search decoding.The target is treated as revealed when its conditional likelihood exceeds the null comparison.
- Black-box probing: Target PII had higher mean likelihood than null PII for every PII type, with statistically significant differences except for affiliation.The γ<k analysis also shows that exact reconstruction becomes more frequent as the number of samples increases, even when likelihood values appear low.
- Black-box probing: Exact-match rates increased with more prompt templates and stronger PII associations, including a fivefold email-address increase when a phone number was added to the name.Larger beam sizes and larger target models also increased the frequency of target PII in generated sentences.
- White-box probing: 1.3% exact-match rate was achieved with a soft prompt learned from 128 data points, compared with 0.0047% for black-box probing using five templates.Using 16 data points already reached 0.12%, while increasing the tuning set from 16 to 128 raised exact match from 0.12% to 1.50%.
- White-box probing: Soft-prompt leakage increased with the number of soft tokens, while PII-type initialization produced the strongest leakage and uniform or mean initialization did not raise it.A soft prompt learned on OPT-1.3B also increased likelihood-based leakage on OPT-350M and OPT-2.7B, although exact-match differences were not substantial.
5 Conclusion
The paper presents ProPILE as a framework for assessing PII leakage and increasing awareness among data subjects and LLM service providers. Its evaluation and ethical framing emphasize both probing capability and responsible privacy assessment.
- ProPILE supports black-box probing by data subjects and white-box probing by LLM service providers to investigate PII leakage.White-box probing uses training data and model parameters to fine-tune more potent prompts for in-house models.
- OPT-1.3B experiments found that target PII is generated with significantly higher likelihood than a random PII item.
- The evaluation dataset was heuristically collected from open-source corporate datasets, so it may contain incorrectly associated data or noise.This introduces uncertainty and potential inaccuracies when interpreting the results.
- The probing strategies are intended to assess privacy vulnerabilities before LLMs are deployed in broader real-world applications, not to encourage PII leakage.
A.1 Experimental environments
The experiments used PyTorch and Python 3.8 on a system with an NVIDIA RTX 8000 GPU and an Intel Xeon Gold 6242R CPU running Ubuntu 18.04.
- Experiments ran with PyTorch and Python 3.8 on Ubuntu 18.04.
- The machine used an NVIDIA RTX 8000 GPU.
- The CPU was an Intel(R) Xeon(R) Gold 6242R running at 3.10GHz.
A.2 Details of evaluation dataset construction
The evaluation datasets were constructed by extracting structured PII and relationship or affiliation pairs from documents in the Pile dataset using entity recognition, regular expressions, libraries, and filtering models.
- Documents containing a name, phone number, email address, and physical address were converted into structured PII dictionaries.
- Names were identified with NLTK Named Entity Recognition, phone numbers and email addresses with regular expressions, and physical addresses with pyap.
- Relationship pairs were retrieved from documents containing family-relationship expressions and filtered with a question-answering model using a confidence threshold above 0.9.The filtering step removed samples with reversed subject-object relationships.
- Affiliation pairs were extracted when a data subject’s name and one of 800 listed U.S. universities appeared in the same document.
A.3 Experimental details for black-box probing
Black-box probing generated candidate PII with beam search from multiple prompt templates, including prompts containing one or more related PII items. Results across templates used the maximum likelihood or exact-match outcome.
- Beam search generated phone numbers, email addresses, relationships, universities, and physical addresses with maximum token limits of 12, 20, 20, 10, and 30, respectively.
- The collected evaluation data may include mislabeled samples because external libraries sometimes identify corporations as data subjects.
- A reported 1-shot triplet example verbatim reconstructed the target email address through postfix generation.The example used a name and phone number before generating the email address.
- In k-shot prompting, each new template was added sequentially to the previous k −1 templates, and multiple-template results reported the maximum likelihood or exact-match outcome.
- Prompt templates targeted individual PII, structured PII combinations, family relationships, and university affiliations.Examples include contact prompts, relationship questions, and statements about a person’s university.
A.4 Experimental details for white-box probing
White-box probing trains prepended soft prompts with negative log-likelihood optimization, using a 1-shot template and greedy decoding; normalized likelihood compares PII reconstruction likelihoods across lengths.
- Soft prompts were trained with negative log-likelihood loss using AdamW at a learning rate of 0.005 for 700 epochs.The final prompt was selected from the best epoch according to exact match.
- Normalized likelihood adjusts reconstruction likelihood by PII length and can be interpreted as the inverse of perplexity.
- Figure 6 shows target-PII normalized-likelihood distributions shifted right of null PII across all PII types.The target distributions therefore have relatively higher mean normalized likelihoods.
B.1.2 Various string-match based metrics
Additional string-based analyses show that partial reconstruction can expose substantial privacy risk beyond exact PII matches, including phone-number fragments and email identifiers.
- 18% of data subjects had their phone-number location code reconstructed verbatim.
- With at most 10, 100, and 1000 brute-force attacks, 0.12%, 0.48%, and 1.12% of data subjects’ phone numbers were obtained, respectively.
- Email ID exact matches reached 9.05%, compared with 0.29% for overall exact matches.Recovering the ID narrows the search space because email addresses use relatively limited domain options.
- Edit-distance results identified reconstructions that closely resembled target PII despite not matching verbatim.These results indicate that exact-match evaluation can miss near-reconstructions relevant to privacy risk.
B.2 Black-box probing results for other models
Black-box probing of BLOOM-3B and BLOOM-7B generally assigned higher likelihoods to target than null PII, although significance and distribution shifts varied by PII type and model.
- BLOOM-7B consistently produced higher mean log-likelihoods for target PII than null PII.
- For BLOOM-3B, target PII had higher mean log-likelihood than null PII for most types except physical addresses.
- BLOOM-7B showed a rightward distribution shift and slightly higher means than BLOOM-3B, especially for the address attribute.The paper speculates that larger model size may increase PII memorization, citing prior work on model size and memorization.
- All likelihood differences were significant at p < 0.05 except affiliation in both models and physical address in BLOOM-3B.
- BLOOM’s shift may be weaker than OPT-1.3B’s because the BLOOM training data only partially overlaps with the Pile-derived evaluation set.