Source-linked AI summary
Locally Differentially Private Document Generation Using Zero Shot Prompting
Saiteja Utpala, Sara Hooker, Pin Yu Chen
TL;DR
Text can reveal personal attributes and authorship through explicit details or linguistic patterns. DP-Prompt uses zero-shot prompting with pretrained language models to paraphrase private documents, and the authors report improved utility alongside reduced deanonymization. They also identify a remaining risk of exposing explicit personal information, particularly in messages or emails.
Problem
Online text can disclose user attributes directly or through linguistic patterns that facilitate authorship attribution.
Source: §1 · p. 1Method
DP-Prompt uses zero-shot prompting of pretrained language models to paraphrase private documents and release the paraphrases as sanitized documents.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors position DP-Prompt as a simple privacy safeguard, particularly relevant as demand for on-device large language models grows.
Source: §1 · p. 2, §7 · p. 9The approach may still reveal explicit personal information, especially in messages or emails compared with online reviews.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Numerous studies have highlighted the privacy risks associated with pretrained large language models. In contrast, our research offers a unique perspective by demonstrating that pretrained large language models can effectively contribute to privacy preservation. We propose a locally differentially private mechanism called DP-Prompt, which leverages the power of pretrained large language models and zero-shot prompting to counter author de-anonymization attacks while minimizing the impact on downstream utility. When DP-Prompt is used with a powerful language model like ChatGPT (gpt-3.5), we observe a notable reduction in the success rate of de-anonymization attacks, showing that it surpasses existing approaches by a considerable margin despite its simpler design. For instance, in the case of the IMDB dataset, DP-Prompt (with ChatGPT) perfectly recovers the clean sentiment F1 score while achieving a 46\% reduction in author identification F1 score against static attackers and a 26\% reduction against adaptive attackers. We conduct extensive experiments across six open-source large language models, ranging up to 7 billion parameters, to analyze various effects of the privacy-utility tradeoff.
1 Introduction
The introduction motivates DP-Prompt as a response to privacy leakage from text and the limits of existing paraphrasing defenses. It proposes zero-shot paraphrasing with pretrained language models and reports strong utility and privacy results, alongside experiments on open-source models.
1 Introduction
- Text can reveal user attributes directly or through linguistic patterns that facilitate authorship attribution, as illustrated by the AOL search-data leak.The incident involved detailed search logs released under random user IDs that nevertheless contained identifying information.Source: §1 · p. 1
- Earlier DP defenses add noise at word or sentence level, while a fine-tuned paraphrasing approach relies on annotated data and extensive computing resources.The introduction notes that word-level approaches can disregard contextual information.Source: §1 · p. 2
- The authors identify on-device language models as a particularly relevant setting because text-completion tasks require fewer resources.They report extensive experiments across six open-source models up to 7 billion parameters.Source: §1 · p. 2
- DP-Prompt prompts pretrained language models to paraphrase private documents zero-shot, then releases the paraphrases as sanitized documents.The paper motivates paraphrasing as a defense against deanonymization and zero-shot prompting as a way to use pretrained models without task-specific, expensive fine-tuning.Source: §1 · p. 2
- With ChatGPT, DP-Prompt surpasses current methods in utility at every privacy level while recovering clean sentiment F1 and reducing author-deanonymization accuracy.The paper also reports experiments with six open-source models, up to 7 billion parameters.Source: §1 · p. 2
2 Preliminaries
The preliminaries define privacy frameworks and distinguish attacker settings used to assess deanonymization. They also outline comparison dimensions for prior mechanisms, including privacy level, fine-tuning, and sanitized-document output.
2 Preliminaries
- PureDP bounds the ratio of output probabilities for neighboring inputs that differ in one element by exp(ϵ).The definition applies to every output subset in the mechanism's range.Source: §2 · p. 2
- The related-work table compares mechanisms by privacy level, whether they fine-tune a model, and whether they generate a sanitized document rather than only sanitized embeddings.The caption identifies these as the table's comparison dimensions.Source: Table 1 · p. 3
- Metric-DP extends Pure-DP to data represented in a general metric space, with the probability bound depending on the distance between inputs.The stated bound uses exp(d(D, D′)).Source: §2 · p. 3
- Local differential privacy perturbs data before transmission, accounting for an untrusted data collector or server.The passage introduces a formal PureLDP definition.Source: §2 · p. 3
- The attack models vary by adaptivity and access mode; adaptive attackers can train on sanitized outputs, whereas static attackers lack that access.The four settings distinguish text from embedding access.Source: §2 · p. 3
3 DP Prompt
DP-Prompt combines a private document with a paraphrase prompt and generates a sanitized paraphrase using differentially private text generation. It applies the exponential mechanism to token selection, using each token’s logit as utility.
3 DP Prompt
- At each generation step, DP-Prompt replaces token selection with the differentially private exponential mechanism, using the token logit as its utility.Sequential text generation is treated as selecting tokens one at a time.Source: §3 · p. 4
- DP-Prompt combines a private document with a prompt template to generate a paraphrase that is released as the sanitized document.The procedure uses zero-shot prompting to instruct the language model to paraphrase the document.Source: §3 · p. 4
- The stated procedure takes a language model, private document, prompt template, clipping vector, temperature, and paraphrase-token count, and outputs a sanitized document.The implementation code is provided for HuggingFace.Source: §3 · p. 4
- Under the theorem’s conditions, DP-Prompt satisfies (2n(b2 − b1)/T)-LDP when generating n tokens at temperature T.The language model must not have been pretrained on the private-document distribution, and all final logits must lie between b1 and b2.Source: §3 · p. 4
4 Experiments
The experiments compare DP-Prompt with ChatGPT against existing mechanisms and assess open-source models, clipping, and top-k sampling. They report stronger utility at matched empirical privacy for ChatGPT, while the open-source results vary by model and sampling choice.
4.2 DP-prompt with ChatGPT (gpt-3.5)
- At a chosen empirical privacy value, DP-Prompt with ChatGPT (GPT-3.5) has significantly higher utility than all baselines, while sentiment F1 remains unchanged and author-identification scores decrease.For embedding-level attackers, author-identification scores decrease by 46% and 25% in IMDB and 53% and 29% in Yelp; for text-level attackers, they decrease by 54% and 10% in IMDB and 73% and 24% in Yelp.Source: Figure 4 · p. 5, §4.2 · p. 6
4.1 Experiment Setup
- The evaluation uses IMDB and Yelp reviews, measuring empirical privacy with author-identification F1 and utility with sentiment F1; static and adaptive attackers use different training and test data.The experiments compare six baselines with DP-Prompt and ChatGPT; baselines are run three times, with mean author-identification F1 and a 2σ band for sentiment F1.Source: §4.1 · p. 5, §4.2 · p. 5
4.4 Effect of clipping logits
- For StableLM-Tuned(7b) on IMDB at temperature 0.75, clipping changes sentiment F1 from 0.71 to 0.65 and author-identification F1 from 0.53 to 0.45.The authors recommend clipping when higher privacy is required and no clipping when higher utility is desired.Source: Table 2 · p. 7, §4.4 · p. 7
4.3 DP-Prompt with open source models
- Open-source base models perform worse than instruction/RLHF-tuned models; StableLM-tuned (7b) improves over 3b, but ChatGPT alone recovers clean sentiment F1 at temperature 1.5.Flan-T5(3b) outperforms StableLM-tuned (7b), while none of the open-source models matches clean sentiment F1 even at temperature 0.75.Source: §4.3 · p. 7
4.5 Effect of Top-K sampling
- At temperature 0.75, top-k sampling does not improve utility enough for open-source DP-Prompt to match ChatGPT.For StableLM-Tuned on IMDB at temperature 0.7 and k=40, sentiment F1 rises from 0.32 to 0.64 while author-identification F1 rises from 0.05 to 0.27.Source: Table 3 · p. 8, §4.5 · p. 8
5 Issue of Data Memorization
The authors discuss possible pretraining exposure to review data and describe how their evaluation addresses that concern. They note that their assumption may not hold if models were exposed to the datasets.
5 Issue of Data Memorization
- The authors acknowledge that ChatGPT and open-source models may have been exposed to online reviews, so the assumption in Theorem 1 may not hold.They state that no annotated paraphrasing data is available for IMDB or Yelp and that the models were not explicitly fine-tuned for paraphrasing those datasets.Source: §5 · p. 8
- The authors argue that their results are likely to generalize to a large extent because the models were not explicitly fine-tuned for IMDB or Yelp paraphrasing.They also note that similar exposure questions apply to word- and sentence-level methods using pretrained embeddings or transformers.Source: §5 · p. 8
- Their evaluation combines sentiment F1 with empirical deanonymization attacks, including adaptive attackers, rather than relying on utility or theoretical privacy alone.The authors note that copying reviews can preserve sentiment utility while failing empirical privacy, and that superficial alterations may fail against adaptive attackers.Source: §5 · p. 8
6 Related Work
Related work spans document privatization at the word, sentence, and document levels, alongside adversarial methods and private model training. DP-Prompt is distinguished by generating sanitized documents with pretrained models through zero-shot prompting rather than fine-tuning.
6 Related Work
- Word-level mechanisms perturb word embeddings or select replacements, but are described as limited by their disregard for contextual information.MadLib adds Laplace noise to embeddings; Mahalanobis and TEM modify the metric or selection mechanism, and Cus-Text customizes vocabulary mappings.Source: §6 · pp. 8–9
- Sentence-level methods add noise to sentence embeddings or maximize Tukey depth, while document-level methods fine-tune GPT-2 or BART for paraphrasing.DP-Prompt instead uses pretrained models and zero-shot prompting rather than resource-intensive fine-tuning.Source: §6 · p. 9
- Adversarial-learning and data-poisoning approaches typically require a surrogate classifier, whereas DP-Prompt requires neither classifier access nor fine-tuning.Source: §6 · p. 9
- Differentially private model training aims to protect models trained on private data; this paper instead generates sanitized documents from private documents using pretrained models.Source: §6 · p. 9
7 Conclusion
DP-Prompt generates sanitized documents by prompting large language models to paraphrase private documents. The paper reports improved utility over current methods across required privacy levels and presents the approach as a defense against de-anonymization.
7 Conclusion
- DP-Prompt prompts large language models to paraphrase private documents, then releases the paraphrases as sanitized documents.The paper describes the mechanism as locally differentially private and simpler than existing methods.Source: §7 · p. 9
- Across extensive experiments, the approach achieves significantly improved utility compared with current methods for any required level of privacy.The conclusion also describes it as a robust defense against de-anonymization attacks.Source: §7 · p. 9
8 Limitations
The study examines zero-shot LLM paraphrasing as an initial step toward document sanitization, but style concealment may not prevent disclosure of explicit personal information. The authors identify further privacy safeguards and evaluation questions for future work.
8 Limitations
- Zero-shot paraphrasing may conceal author writing style while still revealing explicit personal information, including ZIP codes, bank details, or gender.The authors note that naive prompting can leave this risk, particularly for messages or emails compared with online reviews.Source: §8 · p. 9
- Future work includes prompting models to replace defined sensitive attributes with “X,” and investigating hallucinations, prompt-template effects, and stronger text-level privacy attacks.Source: §8 · p. 9
A Proof of Privacy
The privacy proof bounds the output privacy of DP-Prompt under conditions on the model’s logits and generation temperature. The appendix also documents the implementation and auxiliary-data choices for the evaluated mechanisms.
A Proof of Privacy
- With no pretraining on the private-document distribution and bounded logits, the theorem states an output guarantee of (n(b2 − b1)/T)-LDP.The stated conditions bound each logit between b1 and b2; generation uses temperature T for n tokens.Source: A Proof of Privacy · pp. 12–13
- The proof invokes sequential composition and sets ε = 2n(b2 − b1)/T.Source: A Proof of Privacy · p. 13
B Code
- The appendix points to Listing 1 for the Hugging Face implementation of DP-Prompt.Source: B Code · p. 13
C.1 Word-Level Mechanisms
- Word-level mechanisms use 50-dimensional GloVe embeddings and an Annoy indexer with 500 trees for approximate nearest-neighbor search.Source: §C.1 · p. 13
C.2 Sentence-Level Mechanisms
- Truncated-Laplace and Deep-Candidate use additional public data; Deep-Candidate trains a sentence recoder, while Truncated-Laplace sets truncation boundaries.The recoder uses a three-layer MLP, 50 clusters, and 100 random projections; sentence embeddings are 768-dimensional.Source: §C.2 · p. 13
C.3 Document-Level Mechanisms
- The document-level paraphrasing mechanism fine-tunes GPT-2 XL on PAWS, while DP-Prompt and Truncated-Laplace use the same additional data for logit clipping.The clipping thresholds use minimum and maximum values, which are applied to logits before temperature scaling.Source: §C.3 · p. 13
C.4 Static and Adaptive Attacker Architecture
The appendix describes embedding- and text-level attacker architectures and the expected F1 calculations used for random-classifier comparisons. It also identifies the scope of several reported comparisons without providing their numerical outcomes in these passages.
C.4 Static and Adaptive Attacker Architecture
- Embedding-level attackers use a three-layer MLP with 768 hidden units, while text-level attackers fine-tune BERT-base-cased for three epochs.The embedding model uses dropout 0.5; the text-level setup uses batch size 16 and a starting learning rate of 5 × 10^-5.Source: §C.4 · p. 13
C.6 Expected F1 score of random classifier
- For random-classifier evaluation, the appendix derives F1 measures for binary sentiment classification and macro-averaged multi-class author identification.The random classifier predicts each class with probability 1/ℓ; p_i denotes the fraction of documents with label y_i.Source: §C.6 · pp. 14–15
- Figure 6 compares GloVe dimensions 50 and 300 across Madlib, Mahalanobis, and TEM against embedding-level attackers on IMDB.Tables 4 and 5 are identified as Bert Score results across sampling temperatures for ChatGPT and open-source models, respectively.Source: Figure 6 · p. 15, Table 4 · p. 15, Table 5 · p. 15
D Additional Results
The appendix reports no significant benefit from using 300-dimensional rather than 50-dimensional GloVe embeddings in the specified IMDB comparison. It also describes a BERTScore-based assessment of review-paraphrase similarity.
D.1 Comparing 50 and 300-Dimensional Glove Embeddings for Word-Level Mechanism
- No significant benefit was found for 300-dimensional GloVe embeddings over 50-dimensional embeddings on IMDB against embedding-level attackers.The comparison concerns word-level mechanisms and is shown in Figure 6.Source: §D.1 · p. 15
D.2 BERT score for paraphrasing models
- BERTScore with RoBERTa-large is used to assess similarity between each review and its paraphrase.The passage notes that BERTScore has been demonstrated to correlate with human judgments.Source: §D.2 · p. 15
- ChatGPT assessment results are presented in Table 4, while open-source model results are presented in Table 5.Source: §D.2 · p. 15