PaperlayerView exact revision on arXiv

Source-linked AI summary

Differential Privacy for Text Analytics via Natural Text Sanitization

Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, Sherman S. M. Chow

arXiv:2106.01221v1cs.CLcs.CR

TL;DR

Sensitive training texts and language-model attacks make privacy-preserving NLP important, while representation-based approaches can be hard to inspect and use. The paper sanitizes tokens under a new privacy notion and adapts BERT training to sanitized text; experiments report utility and privacy preservation without increased inference-attack success.

  • Problem

    NLP training data can expose sensitive text, while representation-based defenses are hard for users to inspect and may require changes to existing pipelines.

    Source: §1 · p. 1

  • Method

    The paper proposes token-wise sanitization under UMLDP and uses sanitized texts in sanitization-aware BERT pretraining and downstream fine-tuning.

    Source: §1.2 · p. 2, §4.1 · p. 4

  • Results

    The pipeline preserves utility and privacy across three representative NLP tasks without increasing inference-attack success; on SST-2, SANTEXT improves defense rate by 20% with a 4% utility loss.

    Source: §1.2 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Sanitization-aware pretrained BERT models gain 2% absolute average accuracy over publicly available BERT fine-tuning on SST-2 and QNLI.

    Source: §5.4 · p. 8

    Sophisticated sensitivity measures, including context-specific references or personal-information detection, remain future work.

    Source: §4.5 · p. 6, §6 · p. 9

Abstract

from arXiv · show

Texts convey sophisticated knowledge. However, texts also convey sensitive information. Despite the success of general-purpose language models and domain-specific mechanisms with differential privacy (DP), existing text sanitization mechanisms still provide low utility, as cursed by the high-dimensional text representation. The companion issue of utilizing sanitized texts for downstream analytics is also under-explored. This paper takes a direct approach to text sanitization. Our insight is to consider both sensitivity and similarity via our new local DP notion. The sanitized texts also contribute to our sanitization-aware pretraining and fine-tuning, enabling privacy-preserving natural language processing over the BERT language model with promising utility. Surprisingly, the high utility does not boost up the success rate of inference attacks.

1 Introduction

NLP models create privacy risks because training texts can be sensitive and language models enable inference attacks. The paper proposes direct text sanitization and adapts pretraining and fine-tuning to use sanitized texts.

1 Introduction

  • Training texts can expose sensitive information, and language models enable attacks that recover text or reveal information from embeddings.Naïve removal of common personally identifiable information can fail because innocuous fields may be linked with other sources for reidentification.Source: §1 · p. 1

1.1 Sanitizing Sensitive Texts, Naturally

  • Directly sanitizing documents keeps the released data human-readable and requires minimal changes to NLP pipelines, while extending protection to user queries.Users can sanitize queries such as medical histories before sending them to a service provider.Source: §1.1 · pp. 1–2
  • At the same privacy level, the method gains 28% average SST-2 accuracy over Feyisetan et al. (2020) and runs about 60× faster, including precomputation.The comparison is with the state-of-the-art LDP solution producing sanitized documents.Source: §1.1 · p. 2

1.2 Privacy-Preserving NLP, Holistically

  • Sanitization-aware pretraining sanitizes public texts, masks them, and trains the model to predict their original tokens before downstream fine-tuning.The paper reports that this design slightly increases accuracy while additionally ensuring privacy.Source: §1.2 · p. 2
  • Across three representative NLP tasks, the pipeline preserves utility and privacy; on SST-2, SANTEXT improves defense rate by 20% with a 4% utility loss.The sanitization-based pretraining uses 1/6 of the original BERT pretraining data and maintains privacy comparable to original BERT.Source: §1.2 · p. 2

2 Related Work

Prior work addresses privacy risks in language models through attacks, noisy representations, and private representations. The paper instead focuses on releasing usable sanitized text for general-purpose NLP pipelines.

2 Related Work

  • Embedding and training-data attacks can recover sensitive attributes or verbatim text, motivating defenses with rigorous privacy guarantees.The cited work includes a black-box attack extracting training text from GPT-2.Source: §2 · p. 3
  • Adding DP noise to high-dimensional text representations can harm the privacy–utility balance, and those representations are neither human-readable nor directly usable in existing NLP pipelines.The paper relates the difficulty to the large noise magnitude needed across embedding dimensions.Source: §2 · p. 3
  • The proposed approach samples token substitutions from a precomputed distribution to produce natively usable sanitized text, avoiding embedding-space dimensionality and improving efficiency.A concurrent work also considers the whole NLP pipeline but retains the token-projection approach.Source: §2 · p. 3
  • Adversarially learned private representations share the goal of protecting information, but this paper releases sanitized text for general-purpose use instead.Prior representation methods train an adversary to infer sensitive information while optimizing the main model against that adversary.Source: §2 · p. 3

3 Defining (Local) Differential Privacy

The paper models local text sanitization as token replacement under a threat model in which users do not trust parties that can access sanitized documents. It reviews LDP, MLDP, and ULDP before defining UMLDP to account for both sensitivity and similarity.

3 Defining (Local) Differential Privacy

  • Each user replaces tokens in a document locally and shares the sanitized document with a service provider; the threat model treats other parties with access to it and auxiliary information as attackers.A token may be a character, subword, word, or n-gram.Source: §3 · p. 3

3.1 (Variants of) Local Differential Privacy

  • Standard LDP protects all input pairs uniformly, which can reduce utility because unrelated tokens must have similar output distributions; MLDP scales indistinguishability by input distance, while ULDP distinguishes sensitive inputs.For text, the distance metric is intended to capture semantic relationships.Source: §3.1 · pp. 3–4

3.2 Our New Utility-optimized MLDP Notion

  • UMLDP combines sensitivity-based protection with distance-scaled privacy bounds for protected outputs and a separate condition for unprotected outputs.The definition partitions vocabulary into sensitive and non-sensitive inputs and protected and unprotected outputs.Source: §3.2 · p. 4, §3.1 · p. 4
  • For UMLDP's invertible case, unprotected outputs must be noise-free and mapped deterministically; the definition also includes an additive privacy bound.UMLDP retains composability and free post-processing as an LDP notion.Source: §3.2 · p. 4

4 Our Privacy-Preserving NLP Pipeline

The pipeline sanitizes text token by token using SANTEXT or SANTEXT+, applying a variant of the exponential mechanism over native text tokens. Its replacement distributions are designed to preserve semantic similarity while accounting for token sensitivity.

4.1 Overview

  • SANTEXT and SANTEXT+ sample token substitutions using a variant of the exponential mechanism over native text tokens, avoiding high-dimensional token embeddings.Replacement suitability is defined by how well the input's semantics are preserved for downstream tasks.Source: §4.1 · p. 4
  • SANTEXT uses Euclidean distance between token embeddings to measure semantic similarity and assigns higher replacement probability to closer tokens.The distance is computed using an embedding model that maps tokens into a real-valued vector space.Source: §4.1 · p. 4
  • SANTEXT+ applies SANTEXT to sensitive tokens and uses a tailored distribution for non-sensitive tokens to provide UMLDP as a whole.When all tokens are treated as sensitive, SANTEXT+ reduces to SANTEXT.Source: §4.1 · p. 4

4 end

SANTEXT converts private documents into sanitized text for NLP over a pretrained language model, with substitutions guided by token-embedding distance. The paper also proposes encoder pretraining and full-pipeline fine-tuning to adapt downstream tasks to sanitized text.

4 end

  • Encoder pretraining on sanitized public text and fine-tuning the full pipeline on sanitized documents are proposed to adapt downstream tasks to sanitization noise.Encoder pretraining is optional if its cost is considered too high; fine-tuning updates both encoder and task layers.Source: §4 · p. 5

4.2 Base Sanitization Mechanism: SANTEXT

  • SANTEXT samples a replacement token so that tokens closer to the input in embedding space are more likely to be selected.The embedding model maps tokens to vectors, and their Euclidean distance measures token distance.Source: §4.2 · p. 5
  • The embedding model is assumed to be trained on a large public corpus and shared by all users.Source: §4.2 · p. 5

8 end

SANTEXT+ protects a selected sensitive-token set, while the paper also specifies privacy guarantees and ways to adapt BERT-based downstream tasks to sanitized text. Its sensitivity heuristic and reliance on available labels set practical scope boundaries.

8 end

  • The sanitized document can be released to the service provider for NLP tasks.Source: §8 · p. 5

4.3 Enhanced Mechanism: SANTEXT+

  • SANTEXT+ marks the top w least-frequent tokens as sensitive and applies SANTEXT to them; for other tokens, it retains the original with probability 1-p or substitutes it.Token frequency offers a simple way to divide the vocabulary into sensitive and non-sensitive sets.Source: §4.3 · p. 5
  • SANTEXT satisfies MLDP, while SANTEXT+ satisfies (VS, VP, ϵ, ϵ0)-UMLDP under the stated conditions.Source: §4.3 · p. 6

4.4 NLP over Sanitized Text

  • Further pretraining the encoder on sanitized public text uses an adapted masked-language-model objective to predict original masked tokens from sanitized context.The objective may risk privacy by learning to recover original tokens or semantics; reported privacy is comparable to original BERT.Source: §4.4 · p. 6
  • Fine-tuning uses sanitized text-label pairs and updates both encoder and task-layer parameters, assuming ground-truth labels are available to the provider.More realistic or complex labeling processes based on sanitized text are left for future work.Source: §4.4 · p. 6

4.5 Definition of “Sensitivity”

  • For w=0.9 on the sentiment dataset, approximately 98% of stop words are deemed non-sensitive; corpus-specific frequency division can also treat “breast cancer” as non-sensitive among patients.Source: §4.5 · p. 6
  • Sensitivity can vary across users; personalized semantic-tag handling is application-specific and omitted from this work.The paper leaves more sophisticated context-specific sensitivity heuristics for future work.Source: §4.5 · p. 6

5 Experiments

Experiments compare sanitization mechanisms across three NLP tasks, evaluate sanitization-aware pretraining, and examine how privacy parameter ϵ affects utility and privacy. SANTEXT+ has the best utility in the reported mechanism comparisons, while sanitization-aware pretraining improves utility without reducing privacy.

5.2 Comparison of Sanitization Mechanisms

  • SANTEXT+ achieves the best utility across the reported cases, while SANTEXT outperforms FBDD at the same privacy level without the curse of dimensionality.The utility comparison covers four mechanisms across three tasks at similar privacy levels using GloVe embeddings.Source: Table 1 · p. 7, §5.2 · p. 7

5.1 Experimental Setup

  • The experiments cover SST-2 accuracy, MedSTS Pearson correlation, and QNLI prediction of whether a document contains an answer.The study uses sanitized data for training and testing; it uses GloVe or BERT vocabularies and starts pretraining from bert-base-uncased on WikiCorpus.Source: §5.1 · p. 7
  • SANTEXT+ performance is plotted over w and p with ϵ fixed at 2.The setup uses GloVe embeddings of dimension 128, three training epochs, and task-specific batch sizes and learning rates.Source: Figure 3 · p. 7

5.2 Comparison of Sanitization Mechanisms

  • SANTEXT and SANTEXT+ take about 2 minutes on SST-2 versus FBDD’s 117 minutes on a 24-core CPU.The authors attribute the speed difference to reusing a precomputed sampling-probability matrix rather than recalculating noise and searching for nearest neighbors each time.Source: §5.2 · p. 7
  • More restrictive privacy settings, with smaller ϵ, lower utility but increase defense against mask-token inference.The authors report substantially improved defense with only a small utility loss at ϵ = 16, but the supplied passage truncates the specific result.Source: §5.3 · p. 8

5.4 Effectiveness of Pretraining

  • Sanitization-aware pretraining yields an average 2% absolute accuracy gain on SST-2 and QNLI without hurting privacy.The reported privacy level is nearly the same as for the original BERT model, and sometimes better.Source: §5.4 · p. 8

5.5 Influence of Privacy Parameter ϵ

  • As ϵ increases, utility and the probability of leaving a token unchanged rise, while effective output support and privacy decline.For BERT embeddings, the authors suggest ϵ = 15 for about 40% probability of replacing each token; median Nx is about 60% and accuracy about 81%.Source: §5.5 · p. 9

6 Conclusion

The paper introduces UMLDP, which considers sensitivity and similarity, and uses it to support natural text sanitization without the curse of dimensionality. It applies sanitization-aware pretraining and fine-tuning across the PPNLP pipeline, reporting promising performance without boosting inference attacks.

6 Conclusion

  • UMLDP considers both sensitivity and similarity, enabling natural text sanitization mechanisms without the curse of dimensionality.The paper identifies protecting sensitive words while sacrificing privacy of non-sensitive, common words as the design focus.Source: §6 · p. 9
  • The PPNLP pipeline builds privacy in at the root through sanitization-aware pretraining and fine-tuning.The paper reports that more sophisticated sensitivity measures could further strengthen the approach.Source: §6 · p. 9
  • The authors report that their PPNLP solution maintains high utility without boosting inference attacks.They describe the solution as discerning between beneficial utility and harmful inference attacks.Source: §6 · p. 9

A Supplementary Formalism Details

The supplementary appendix gives formal details for ULDP and the privacy guarantees, then describes how the sanitization distribution behaves across privacy levels and provides qualitative examples.

A.1 Definition of ULDP

  • Definition 4 frames ULDP over a shared VS = VP subset of vocabulary V, with privacy parameter ϵ and output conditions for y in VP.The definition states that outputs in VP have positive probability for x and zero probability for x′ when x and x′ differ.Source: §A.1 · p. 12

A.2 Differential Privacy Guarantee

  • For a single-token document, the SANTEXT privacy proof relies mainly on the triangle inequality of d.The appendix generalizes to longer documents by sanitizing each token independently.Source: §A.2 · p. 12
  • For documents longer than one token, independent token sanitization leads to a product of token-level output probabilities.The stated generalization uses composability to obtain the privacy bound for arbitrary D and D′.Source: §A.2 · p. 12
  • For x and x′ in VS, SANTEXT+ restricts the output y to VP and uses the same proof as SANTEXT.For x in VN, the probability of remaining unchanged is 1−p; the appendix says these inequalities establish UMLDP properties and extends to longer documents by composability.Source: §A.2 · p. 12

A.3 Qualitative Observations

  • At ϵ = 0, SANTEXT samples uniformly over V independently of x, while as ϵ approaches infinity, x almost always remains unchanged.The appendix characterizes these endpoints as perfect privacy with no information preserved, and no privacy loss with no utility loss, respectively.Source: §A.3 · p. 12
  • For finite positive ϵ, SANTEXT has full support over V and assigns higher probability to outputs closer to x under d.As ϵ increases, outputs concentrate more on nearby tokens, which the appendix associates with better utility but weaker privacy when concentration is excessive.Source: §A.3 · p. 12
  • For SANTEXT+, the observations apply directly when x is in VS; for x in VN, p controls mapping to VP, and larger p strengthens privacy.The appendix attributes this to a smaller probability, 1−p, that x remains unchanged.Source: §A.3 · p. 12
  • Table 4 presents sanitized-text examples from SST-2 and QNLI at different privacy levels using GloVe embeddings.The caption identifies the examples and conditions but does not state an outcome comparison.Source: Table 4 · p. 13

B Qualitative Examples

The appendix presents qualitative examples of sanitized text from SST-2 and QNLI across privacy levels.

B Qualitative Examples

  • Table 4 shows sanitized-text examples from SST-2 and QNLI at different privacy levels based on GloVe embeddings.The caption identifies the example datasets, mechanism outputs, and privacy-level variation.Source: B Qualitative Examples · p. 13

C Supplementary Related Works

The supplementary related work surveys cryptographic and hardware-aided approaches to privacy-preserving analytics, alongside a text-specific hidden-representation method. It describes trade-offs involving computation, communication, and security.

Cryptographic Protection of (Text) Analytics

  • Cryptographic methods such as homomorphic encryption and secure multiparty computation can support NLP tasks but may slow computation by orders of magnitude.The passage also warns that replacing cryptography with heuristics can be error-prone, citing a parameter-recovery attack on oblivious inference.Source: C Supplementary Related Works · p. 13
  • Wagh et al. (2021) use multiple non-colluding servers for secure training, but SMC requires multiple communication rounds.The passage says a dedicated connection among servers is more desirable.Source: C Supplementary Related Works · p. 13
  • Cryptographic approaches mostly exclude membership-inference immunity from their protection scope, while DP can be applied to training data before cryptographic processing.The passage attributes the membership-inference scope point to Shokri et al. (2017).Source: C Supplementary Related Works · pp. 13–14

Supplementary Related Works

Hardware-Aided Approaches

  • Hardware-aided GPU solutions must protect linear and nonlinear operations and integrate those protections; Slalom (Tramèr and Boneh, 2019) supports inference, while Goten (Ng et al., 2021) addresses training.The passage also identifies GForce (Ng and Chow, 2021) as an early GPU-friendly protocol for nonlinear layers.Source: C Supplementary Related Works · p. 14
Loading 2106.01221v1…