Source-linked AI summary
The Limits of Word Level Differential Privacy
Justus Mattern, Benjamin Weggenmann, Florian Kerschbaum
TL;DR
Word-level differential privacy anonymizes text by perturbing individual word embeddings, but the paper identifies theoretical and practical limitations, including weak protection against advanced deanonymization attacks. It proposes transformer-based paraphrasing with a formal privacy guarantee and reports that it outperforms word-level mechanisms in experiments.
Problem
Word-level text anonymization has mathematical privacy constraints and practical shortcomings in deanonymization protection, content preservation, and language quality.
Source: §1 · pp. 1–2Method
The paper fine-tunes GPT-2 to generate paraphrases from sentence pairs whose five SNLI labels all indicate entailment, varying sampling temperature to balance privacy and utility.
Source: §4.1 · p. 6, §4.2 · p. 6Results
The proposed paraphrasing approach evades most identified drawbacks and outperforms word-level mechanisms in the reported experiments.
Source: §1 · p. 2Takeaways & Limitations
Transformer-based paraphrasing offers a text-anonymization alternative that retains a formal privacy guarantee while avoiding most identified weaknesses of word-level methods.
Source: §1 · p. 2Metric privacy can yield seemingly smaller ϵ values without changing the mechanism, while sentence-length growth in the metric can obscure linear privacy-loss growth.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
As the issues of privacy and trust are receiving increasing attention within the research community, various attempts have been made to anonymize textual data. A significant subset of these approaches incorporate differentially private mechanisms to perturb word embeddings, thus replacing individual words in a sentence. While these methods represent very important contributions, have various advantages over other techniques and do show anonymization capabilities, they have several shortcomings. In this paper, we investigate these weaknesses and demonstrate significant mathematical constraints diminishing the theoretical privacy guarantee as well as major practical shortcomings with regard to the protection against deanonymization attacks, the preservation of content of the original sentences as well as the quality of the language output. Finally, we propose a new method for text anonymization based on transformer based language models fine-tuned for paraphrasing that circumvents most of the identified weaknesses and also offers a formal privacy guarantee. We evaluate the performance of our method via thorough experimentation and demonstrate superior performance over the discussed mechanisms.
1 Introduction
Authorship attribution can expose the originators of sensitive text, motivating methods that anonymize writers. The paper examines limits of word-level differential privacy and proposes paraphrasing as an alternative that outperforms those mechanisms in experiments.
1 Introduction
- Authorship attribution can identify text authors, creating potential privacy risks when sensitive data is shared or used by companies and researchers.The paper identifies healthcare and financial services as industries handling personal information.Source: §1 · p. 1
- Prior text obfuscation work either learns anonymous representations for downstream tasks or transforms sentences into human-readable text to remove author-revealing properties.The second category commonly perturbs individual word embeddings under differential privacy and samples nearby replacement words.Source: §1 · p. 1
- The authors investigate word-level anonymization’s theoretical privacy constraints, language quality, and ability to mitigate deanonymization attacks on IMDb and Yelp reviews.The attack experiments use state-of-the-art methods on movie-review and business-review datasets.Source: §1 · p. 1
- Paraphrasing-based anonymization maintains a formal privacy guarantee, evades most identified drawbacks, and outperforms word-level mechanisms in the reported experiments.The paper motivates the approach by experiments in which crowdsourced human rewritings successfully anonymize original authors.Source: §1 · p. 2
2 Background
This background introduces differential privacy and its local and metric variants, then describes the common word-level anonymization pipeline of perturbing embeddings and replacing tokens.
2 Background
- The paper situates word-level embedding perturbation as a common approach among text-anonymization methods.It frames the approach as applying differential privacy to individual words.Source: §2 · p. 2
2.1 Differential Privacy
- Differential privacy calibrates noise to a query’s sensitivity, its maximum change across adjacent datasets.The background identifies L2-sensitivity as the quantity used by the Planar Laplace mechanism.Source: §2.1 · p. 2
- Local differential privacy adds noise at the data source and treats any two inputs as adjacent, which can make a satisfactory privacy-utility trade-off difficult.This contrasts with the central model, which assumes records are collected in a central database.Source: §2.1 · p. 2
2.1.1 Generalization with metrics
- Metric privacy bounds the indistinguishability level between two inputs by ϵ times their distance, generalizing the privacy definition to metric spaces.Inputs are adjacent under the metric when their distance is at most 1.Source: §2.1.1 · p. 2
2.2 Word perturbations for anonymization
- Word-level anonymization maps each token to a pretrained embedding, adds sampled noise, and replaces the token with a word near the perturbed embedding.The framework applies this process token by token and states that it fulfills ϵd-privacy for its sentence distance metric.Source: §2.2 · p. 3
3 Limitations of word level privacy
Word-level privacy mechanisms are computationally simple and dataset-independent, but this section identifies theoretical constraints and language-generation limits. Their guarantees can be length-sensitive, while token-level changes restrict grammatical consistency and syntactic variation.
3 Limitations of word level privacy
- Word-level DP methods use a simpler approach with lower computational expense and can be deployed across datasets without retraining for new authors.The section examines their theoretical and language-output shortcomings.Source: §3 · p. 3
3.1 DP related constraints
- Because word-level mechanisms preserve input length, they limit guarantees to fixed-length sequences, restricting sentence scope and expressiveness.Different-length inputs make the relevant outcome-set probabilities contradict pure DP; approximate DP would require δ = 1.Source: §3.1 · p. 3
- For an n-token sequence, the total privacy budget may grow linearly with length, reaching nε under the stated independent-token analysis.Metric privacy can conceal this growth when its distance increases with sentence length.Source: §3.1 · p. 3
- Local DP treats any two inputs as adjacent, a strong condition that can make obfuscated data questionable for analysis.It typically adds substantial noise and may prompt use of a larger ε, which can also permit attackers to infer sensitive information.Source: §3.1 · p. 4
- Metric privacy changes how ε is interpreted, not the mechanism, and can relax rather than resolve local-DP concerns.A length-growing metric can make ε appear smaller while absorbing privacy loss, even when writing style is nearly unchanged.Source: §3.1 · pp. 3–4
3.2 Language constraints
- Independent token perturbations make consistency difficult, so replacing a word with another word type can induce grammatical errors.The authors report that grammatical errors tend to increase as the privacy budget shrinks.Source: §3.2 · pp. 4–5
- At ε = 8, 17.3% of word types changed; at ε = 10, 7.8% changed under Feyisetan et al.’s mechanism.The paper estimates these rates using WordNet labels on perturbed tokens; Figure 1 presents the percentage of word-type changes.Source: §3.2 · p. 5, Figure 1 · p. 5
- Word-level mechanisms lack syntactic changes, relying mainly on lexical substitutions rather than changing word order or sentence length.This limits linguistic variety and leaves syntactic features available as author-identifying clues.Source: §3.2 · p. 5
4 Anonymization through paraphrasing
The paper proposes transformer-based paraphrasing as an alternative to word-level anonymization and develops a temperature-controlled sampling method with a local differential privacy guarantee. Its privacy budget grows with output length, while the approach avoids some language and fixed-length constraints of earlier mechanisms.
4 Anonymization through paraphrasing
- Fine-tuning a large language model for paraphrasing is proposed as an alternative to word-level perturbations to address their identified weaknesses.The approach is motivated in part by crowdsourced human rewritings that successfully anonymized authors (Almishari et al., 2014).Source: §4 · p. 5, §4.1 · pp. 5–6
4.1 Generating paraphrases
- GPT-2 is fine-tuned on SNLI sentence pairs for which all five crowdsourced labels indicate semantic entailment.This imitates human rewriting while using a large-scale transformer language model.Source: §4.1 · p. 6
4.2 Balancing privacy and utility
- Higher sampling temperature smooths word probabilities and changes sentence variability, providing a control for balancing privacy and utility.The experiments vary temperature during text sampling to evaluate this effect.Source: §4.2 · p. 6
A discrete version of the Exponential mechanism
- At each generated token, softmax sampling is an Exponential mechanism with ϵ = 2∆q/T; bounded logits give ∆q ≤ 1, while the total budget is ϵ·n.The model is locally differentially private, but its budget may grow linearly with output length; it avoids earlier language and fixed-output-length constraints.Source: A discrete version of the Exponential… · pp. 6–7
5 Evaluation
The evaluation compares word-level perturbation methods with paraphrasing on IMDb and Yelp, measuring privacy, sentiment utility, semantic similarity, and language quality. Paraphrasing performs better against adaptive adversaries and, at comparable author-classifier protection, preserves semantics and language quality better.
5 Evaluation
- The evaluation compares Laplace-noised GloVe, hyperbolic-noised Poincaré, and paraphrasing using state-of-the-art methods in static and adaptive attack settings.Prior evaluations often used weaker attackers or static training on original data alone.Source: §5 · p. 7
5.1 Evaluation metrics
- Evaluation uses static and adaptive BERT classifiers with MCC, relative gain for the author–sentiment trade-off, SBERT cosine similarity, and GPT-2 perplexity for language quality.Relative gain is defined as normalized sentiment MCC minus normalized author MCC.Source: §5.1 · p. 7
5.2 Implementation Details
- The word-level baselines use 50-dimensional GloVe and Poincaré embeddings; the limited Poincaré vocabulary leaves out-of-vocabulary words unperturbed, violating DP.GPT-2 and BERT are pretrained checkpoints fine-tuned on a single NVIDIA T4 GPU.Source: §5.2 · p. 8
5.3 Datasets
- IMDb and Yelp subsets retain the ten most prolific users, yielding 10,000 and 15,729 reviews, with positive sentiment thresholds of 5 and 3, respectively.The ratings use scales of 1–10 for IMDb and 1–5 for Yelp.Source: §5.3 · p. 8
5.4 Results
- Paraphrasing significantly outperforms word-level mechanisms against adaptive adversaries and, at comparable author-classifier protection, provides better semantic preservation and language quality.The comparison uses MCC for authorship and sentiment classifiers; Table 1 reports classifier performance and relative-gain trade-offs.Source: §5.4 · pp. 8–9, Table 1 · p. 8
6 Related work
Related work spans differentially private vector representations, human-readable text obfuscation, non-DP authorship obfuscation, and privacy-preserving model training. The paper’s inference-time protection is presented as complementary to methods that protect training data.
Other DP mechanisms for text
- Some DP text methods output bag-of-words or dense vectors, while others generate human-readable text using encoder-decoder or sequence-to-sequence models.The cited examples include Bo et al. (2021) and Weggenmann et al. (2022).Source: §6 · p. 9
Authorship obfuscation without DP
- Non-DP authorship obfuscation includes rule-based text edits, discriminator-based deep learning, and private text-vector representations.Examples include synonym replacements and word removals.Source: §6 · p. 9
Differentially private optimization
- DP-SGD and related methods protect model training data, complementing this paper’s approaches that protect data during inference.The cited work includes generative and discriminative language models trained with differentially private optimization.Source: §6 · p. 9
7 Conclusion
The paper identifies weaknesses in word-level differential privacy mechanisms and proposes paraphrasing as an alternative. It reports better protection against adaptive adversaries, while word-level mechanisms should be favored against weaker adversaries.
7 Conclusion
- The proposed paraphrasing model circumvents most of the weaknesses identified in word-level differential privacy mechanisms.Source: §7 · p. 9
- Paraphrasing outperforms word-level mechanisms against adaptive adversaries, while word-level mechanisms should be favored against weaker adversaries.Source: §7 · p. 9
- Future work could integrate auxiliary adversarial losses into paraphrasing systems or better preserve the source text’s core information.Source: §7 · p. 9
8 Ethical Considerations
The ethical considerations cover potential misuse of anonymization, bias in the language model, and uncertainty about reproducing comparison systems. They identify risks involving harmful or deceptive content and possible changes to anonymized authors’ statements.
8 Ethical Considerations
- Anonymization could be used to spread hate speech, fake reviews, or fake news, alongside its intended privacy and freedom-of-speech purposes.Source: §8 · p. 9
- GPT-2’s demographic biases could cause unforeseen effects in generated text and potentially alter anonymized authors’ statements.Source: §8 · p. 9
- Because comparison systems lacked public code and implementation details, the authors cannot guarantee exact replication of those systems.Source: §8 · p. 10
A.1 Information about terms of use for data
This appendix section provides dataset terms-of-use and licensing information. It reports differing access and license conditions for WordNet, WebIsADB, IMDb, and Yelp.
A.1 Information about terms of use for data
- WordNet is accessible online without a specific access request and is available for research and commercial use under its 3.0 license.Source: §A.1 · p. 13
- WebIsADB is accessible online without a specific access request and is licensed under Creative Commons Attribution-Non Commercial-Share Alike 3.0.Source: §A.1 · p. 13
- The authors could not find license specifications for IMDb movie reviews, which are available online without a specific access request.Source: §A.1 · p. 13
- Researchers using the Yelp dataset must sign its terms of use.Source: §A.1 · p. 13