Source-linked AI summary
Selective Differential Privacy for Language Modeling
Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, Zhou Yu
TL;DR
Language models can memorize sensitive training text, while traditional DP protects all tokens even when private information is sparse. The paper introduces selective differential privacy and Selective-DPSGD for RNN language models, reporting better utility than traditional DP and greater robustness to attacks than models without DP protection across two tasks.
Problem
Traditional DP protects every token in a record, although natural-language data may contain only sparse private information, and this can lead to poor model performance.
Source: §1 · pp. 1–2Method
Selective-DP uses a policy function to identify sensitive attributes, and Selective-DPSGD applies the resulting protection to RNN-based language models.
Source: §1 · p. 2Results
Across WikiText-2 and a synthetic customer-service dialog dataset, S-DP models achieve better utility than traditional DP and are more robust under various attacks than models without DP protection.
Source: §7 · p. 9Takeaways & Limitations
S-DP could also apply beyond NLP in domains where only partial data require protection.
Source: §7 · p. 9Building better policy functions is outside this paper’s scope, and improvements to them are compatible with S-DP.
Source: §5 · p. 5
Abstract
from arXiv · showhide
With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete application--dialog systems. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities while remaining safe under various privacy attacks compared to the baselines. The data and code are released at https://github.com/wyshi/lm_privacy to facilitate future research .
1 Introduction
Language models trained on sensitive text can memorize it, while protecting every token with traditional DP can impair utility. The paper proposes selective differential privacy and Selective-DPSGD to protect sensitive attributes while improving utility, and evaluates the approach on language generation and dialog systems.
1 Introduction
- Traditional DP protects each data point as a whole, although language data may contain only sparse private information, such as an SSN token.The paper notes that protecting an entire sentence can make it harder to learn its underlying language pattern [McMahan et al., 2018].Source: §1 · p. 1
- Selective-DP protects sensitive attributes in a training record while allowing non-sensitive attributes to remain unprotected.A user-defined policy function specifies which dimensions are sensitive for an application.Source: §1 · p. 2
- Selective-DPSGD applies regular SGD to non-private variables and DPSGD to private variables in RNN-based language models.The method also batches private and public tokens together for alternating updates.Source: §1 · p. 2
- Selective-DPSGD improves utility while remaining safe under popular privacy attacks in language generation and dialog-system experiments.The experiments report model utility and theoretical privacy guarantees alongside empirical robustness to attacks.Source: §1 · p. 2
- The paper proposes a dialog dataset and reports improved utility and attack safety on language generation and dialog systems.It also discusses imperfect policy functions and presents preliminary results for contextual policies and Transformer models.Source: §1 · p. 2
2 Related Work
Prior work applies differential privacy and related protections to language data, but the paper distinguishes its approach by protecting sensitive attributes within a record rather than all examples uniformly.
2 Related Work
- Language-model attacks can infer secrets in training data, motivating privacy mechanisms for models trained on sensitive text (Carlini et al., 2019).The related-work discussion identifies language models as vulnerable to attacks.Source: §2 · p. 2
- Prior approaches include DPSGD and PATE, as well as personalized and one-sided DP; this paper instead targets sensitive attributes within records.The passage cites DPSGD (Abadi et al., 2016), PATE (Papernot et al., 2018), personalized DP, and one-sided DP (Doudalis et al., 2017).Source: §2 · p. 2
- Unlike work applying DPSGD or text de-identification, this method protects only sensitive portions in centralized learning and can also be adapted to federated learning.The cited examples include DP-FedAvg and DP-FedSGD (McMahan et al., 2018) and probabilistic de-identification (Adelani et al., 2020).Source: §2 · pp. 2–3
- Selective-DP is complementary to methods that improve private deep learning through training-process or architecture changes, and can be combined with them.The passage cites training-process methods (Wang et al., 2021, 2019; Thakkar et al., 2019; Lee and Kifer, 2020) and architecture methods (Papernot et al., 2020).Source: §2 · p. 3
3 Backgrounds
The background introduces sequence probability modeling and differential privacy. Language models learn sequence probabilities from a corpus, while DP limits what an algorithm’s output reveals about an individual record.
3 Backgrounds
- The language-modeling setup treats text as a sequence of tokens and trains on a corpus of such sequences.Each sequence is denoted x = (x1, x2, . . . , xn).Source: §3 · p. 3
- A language model represents a sequence probability through the chain rule and learns it by minimizing corpus negative log-likelihood.The model is a neural network parameterized by θ, such as an RNN.Source: §3 · p. 3
- Differential privacy aims to prevent an algorithm’s output from revealing much about whether an individual record is in its input dataset.The formal definition considers neighboring datasets differing in one record.Source: §3 · p. 3
4 Selective Differential Privacy
Selective-DP distinguishes sensitive from non-sensitive attributes within a record using a policy function, then limits privacy guarantees to the sensitive portion. Selective-DPSGD realizes this distinction by applying private and regular updates to the corresponding variables.
4 Selective Differential Privacy
- Selective-DP uses a policy function to distinguish private and non-private attributes within a record, addressing a distinction unavailable in canonical DP notions.The paper motivates this design by the potential utility gains when private attributes are sparse, especially in NLP.Source: §4 · p. 3
- Users can define policy functions to encode application-specific privacy rules for entities or contextual attributes.The functions may take forms such as neural networks or regular expressions, and can conservatively mark complete sentences as sensitive.Source: §4 · p. 3
- For language modeling, records are token sequences, and F-neighboring datasets differ by a private attribute in one record.The text gives changing an ID number as a neighboring-dataset example, unlike changing a greeting without private information.Source: §4 · pp. 3–4
- S-DP guarantees indistinguishability for sensitive attributes while placing no constraint on leakage of non-sensitive attributes.The stated guarantee considers an attacker who may know everything except the targeted sensitive attribute.Source: §4 · p. 4
4.1 Selective Privacy Mechanism
- Selective-DPSGD identifies private attributes, maps them to model variables, then applies regular SGD to non-private variables and DPSGD to private variables.It also clips and adds noise to private hidden states because they pass into later regular updates; batching groups private and non-private tuples for alternating updates.Source: §4.1 · p. 4, Figure 2 · p. 4
5 Experiments
The experiments use WikiText-2 for language modeling and a synthetic customer-service dialog dataset, with sensitive content identified by task-specific policies. Selective-DPSGD alternates regular and private updates, and also clips and noises hidden states for private inputs.
5 Experiments
- WikiText-2 is the already-public language-modeling corpus, where the policy function treats every digit as private.The digit detector outputs 0 for digits and 1 for other tokens.Source: §5 · pp. 4–5
- Selective-DPSGD applies regular SGD updates to non-private tuples and private DPSGD updates to private tuples within each sampled batch.The algorithm splits batches using the privacy bit matrix; private gradients are clipped and noised, and private hidden states are also clipped and noised.Source: §5 · p. 5
- CUSTOMERSIM simulates customer-service dialogs with fixed agendas and template utterances, marking names, addresses, phone numbers, orders, and tracking numbers as sensitive via regex.The dataset uses synthetic user information to avoid potential harm from real dialog transcripts.Source: §5 · p. 5
- The quality of policy functions is outside this paper’s scope, although improved policies are compatible with S-DP.Users can specify policies for different sensitive attributes and applications.Source: §5 · p. 5
CUSTOMERSIM
Across WikiText-2 and CUSTOMERSIM, S-DPSGD improves utility over DPSGD at similar privacy budgets, while attack results vary by dataset. The section also compares selective noise with anonymization and examines the risk of imperfect sensitive-information detection.
CUSTOMERSIM
- CUSTOMERSIM uses simulated, fixed-agenda template dialogs and regex detection for names, addresses, phone numbers, orders, and tracking numbers.The models use one-layer LSTMs with 200-dimensional embeddings and hidden states; all private LSTM states are protected, and models are trained five times.Source: CUSTOMERSIM · p. 5, Table 1 · p. 5
5.1 Evaluation
- Evaluation compares perplexity and top-1 next-word accuracy with theoretical privacy budgets and empirical attacks, measuring utility at a given privacy level or vice versa.Table 2 reports model utility and privacy guarantees for WikiText-2 and CUSTOMERSIM.Source: §5.1 · pp. 5–6, Table 2 · p. 6
5.1.1 Attack Details
- Canary insertion measures exposure of inserted secrets, while membership inference tests whether samples belong to training data; random guessing yields 50% accuracy in the described evaluation.The membership-inference set contains 500 protected secrets and 500 similar generated samples; the canary secret is inserted into training data 10 times.Source: §5.1.1 · p. 6
5.2 WikiText-2 Results
- At ϵ=4.91, S-DPSGD reaches average test PPL 160.04 versus DPSGD’s 305.86 at ϵ=4.89 on WikiText-2; No-DP is best at 60.98.At similar utility, S-DPSGD has lower canary exposure than No-DP and, at similar ϵ, lower perplexity than DPSGD while remaining safe to the attacks.Source: §5.2 · pp. 6–7, Figure 3 · p. 7
5.3 CUSTOMERSIM Results
- On CUSTOMERSIM, S-DPSGD achieves better perplexity than DPSGD (10.42 vs. 11.82) at similar ϵ and lower canary exposure given the same utility; membership inference accuracy is only 60+%.The perplexity gap is smaller than on WikiText-2 because CUSTOMERSIM has more sensitive tokens (18.3% vs. 2.8%).Source: §5.3 · pp. 7–8
5.4 Data Anonymization and Selective-DP
- Data anonymization masks sensitive tokens completely, whereas S-DP noises them and offers a tunable privacy-utility trade-off; both depend on a policy function that can miss sensitive information.Source: §5.4 · p. 8
5.4.1 S-DP achieves better utility
- On private tokens, anonymization’s masking prevents learning their patterns and produces worse PPL, while S-DP trades privacy budget ϵ against private-token PPL.The authors say S-DP may retain patterns such as an SSN’s structure without learning its exact value; anonymization offers no parameter to tune this trade-off.Source: Figure 5 · p. 8, §5.4.1 · p. 8
5.4.2 Imperfect Policy Function
- When the policy misses the secret “341752,” exposure rises for both methods as perplexity falls, but S-DP has lower exposure than anonymization around the middle perplexity range.The authors identify imperfect policy functions as a shared risk beyond this paper’s scope; better detection is compatible with S-DP.Source: Figure 6 · p. 8, §5.4.2 · pp. 8–9
6 Preliminary Results on Contextual Policy Function and Transformer
Preliminary WikiText-2 experiments with contextual and noncontextual policies suggest S-DP can improve privacy-preserving language-model utility, including when about one-third of tokens are sensitive.
6 Preliminary Results on Contextual Policy Function and Transformer
- 22.56 and 25.61 perplexity after DPSGD fine-tuning, versus 27.05 for state-of-the-art DP language models at similar privacy budget.The two policy-function settings are reported in that order; the comparison appears in Table 3.Source: Table 3 · p. 9, §6 · p. 9
- The noncontextual policy protects 18 named-entity types (16.4% of tokens), while the contextual policy also protects sentence subjects, objects, proper nouns, and pronouns (34.8%).Redacted-model perplexities were 24.30 and 38.66, respectively; DPSGD fine-tuning on original data reduced them to 22.56 and 25.61.Source: §6 · p. 9
- The results suggest S-DP can boost privacy-preserving language-model utility even when one-third of tokens are treated as sensitive.These are preliminary results on contextual policies and large Transformer models on WikiText-2.Source: §6 · p. 9
7 Conclusions
The paper introduces selective differential privacy to protect sensitive portions of data while improving language-model utility. Experiments on WikiText-2 and synthetic customer-service dialogs report better utility than traditional DP and greater robustness to attacks than models without DP.
7 Conclusions
- Across WikiText-2 and a synthetic customer-service dialog dataset, S-DP models achieve better utility than traditional DP and greater robustness to attacks than models without DP protection.The paper reports theoretical privacy guarantees as well as empirical robustness under attacks.Source: §7 · p. 9
- Selective differential privacy provides rigorous guarantees for sensitive portions of data, and Selective-DPSGD implements it for RNNs.The conclusion presents the method as improving utility while protecting the sensitive portion of the data.Source: §7 · p. 9
- The authors suggest S-DP may apply beyond NLP to domains such as image recognition when only part of the data requires protection.They describe this as a potential application.Source: §7 · p. 9
A.1 Ethical Consideration
The authors use public or synthetic datasets to minimize potential real-world harm and describe the scope of their attack evaluations.
A.1 Ethical Consideration
- The experiments use public WikiText-2 and synthetic CUSTOMERSIM dialogs and personal information; future work plans to continue using public or synthetic datasets.The stated aim is to prevent real-world data leakage.Source: §A.1 · p. 11
- The attacks are standard attacks tailored to the specific tasks, so their generalization and misuse against other language models are described as difficult.Source: §A.1 · p. 11
- The authors plan to release the code so others can access the algorithms and protect their own data.Source: §A.1 · p. 11
A.2 Limitations
The analysis composes private and non-private queries to establish selective privacy guarantees, while experiments examine parameter trade-offs and membership inference on CUSTOMERSIM. The stated limitation is that missed sensitive attributes may not receive privacy guarantees.
A.2 Limitations
- If the policy function misses sensitive attributes, their privacy may not be guaranteed; the authors identify better policies and contextual information as areas for further work.They describe contextual protection as preliminary and plan to refine contextual policy functions.Source: §A.2 · p. 11
A.3 Privacy Analysis
- The analysis models gradient updates and hidden states as queries taking training data x and auxiliary information w, which captures dependence on prior-round model parameters.For bounded-output type-1 queries, adding Gaussian noise can provide (ε, δ)-DP for arbitrary auxiliary input.Source: §A.3 · p. 11
- Theorem 2 states that for a composition of k noisy type-1 or type-2 queries, if the noisy type-1 subset is (ϵ, δ)-DP, then the full composition is (F, ϵ, δ)-S-DP.The paper uses a moment accountant to calculate DP guarantees for the composition of all privacy queries.Source: §A.3 · pp. 11–12
A.4 Models with Different Parameters
- With σ = 0.5, the clipping threshold C strongly affected performance; reducing σ from 0.5 to 0.1 lowered perplexity at a huge privacy-budget cost.The main experiments used σ = 0.5 and C = 1e-3 as a balance between utility and privacy.Source: §A.4 · p. 12, Figure 7 · p. 12
A.5 Membership Inference on CUSTOMERSIM
- Around 58% was the best original membership-inference accuracy on CUSTOMERSIM; an advanced attack reached only slightly above the 60%+ random-guess probability and was unsuccessful.The advanced attack selected 300 names from 1,000 based on its best and worst predictions.Source: §A.5 · p. 12, Figure 8 · p. 12