PaperlayerView exact revision on arXiv

Source-linked AI summary

DP-MLM: Differentially Private Text Rewriting Using Masked Language Models

Stephen Meisenbacher, Maulik Chevli, Juraj Vladika, Florian Matthes

arXiv:2407.00637v1cs.CL

TL;DR

Prior DP text rewriting relies on autoregressive models that lack a way to contextualize private rewriting. DP-MLM uses masked language models with concatenated sentence context to rewrite text privately. The authors report stronger utility on many benchmarks and competitive empirical privacy scores.

  • Problem

    Existing DP text rewriting relies on autoregressive models without a mechanism to contextualize private rewriting, amid a broader challenge of balancing privacy and utility.

    Source: §1 · p. 1

  • Method

    DP-MLM uses masked-token prediction in BERT-based models and input-sentence context to produce differentially private text rewrites.

    Source: §1 · p. 1, §8 · p. 9

  • Results

    DP-MLM outperforms prior state-of-the-art methods on many utility benchmarks and achieves competitive empirical privacy scores.

    Source: §7 · p. 8, §8 · p. 9

  • Takeaways & Limitations

    The authors conclude that encoder-only MLMs can balance utility and privacy in private text rewriting.

    Source: §8 · p. 9

    The study evaluates one representative language model per mechanism, leaving other BERT-based models outside its scope.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The task of text privatization using Differential Privacy has recently taken the form of $\textit{text rewriting}$, in which an input text is obfuscated via the use of generative (large) language models. While these methods have shown promising results in the ability to preserve privacy, these methods rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. In response to this, we propose $\textbf{DP-MLM}$, a new method for differentially private text rewriting based on leveraging masked language models (MLMs) to rewrite text in a semantically similar $\textit{and}$ obfuscated manner. We accomplish this with a simple contextualization technique, whereby we rewrite a text one token at a time. We find that utilizing encoder-only MLMs provides better utility preservation at lower $\varepsilon$ levels, as compared to previous methods relying on larger models with a decoder. In addition, MLMs allow for greater customization of the rewriting mechanism, as opposed to generative approaches. We make the code for $\textbf{DP-MLM}$ public and reusable, found at https://github.com/sjmeis/DPMLM .

1 Introduction

DP text rewriting must balance privacy and utility, while prior rewriting methods use decoder-based generation without studying encoder-only MLMs. DP-MLM uses masked-token prediction and concatenated context to produce private rewrites; the authors report gains over prior methods on many benchmarks.

1 Introduction

  • Prior DP text rewriting uses paraphrase models or prompted LLMs, but the benefits of encoder-only models had not been studied alongside decoder models.The motivating question is how MLMs can support contextually relevant, privacy-preserving rewriting.Source: §1 · p. 1
  • DP-MLM uses masked-token prediction in BERT-based models and incorporates the input sentence's context into private rewriting.The authors report experiments testing meaning preservation and empirical defense against adversarial privacy attacks.Source: §1 · p. 1

1. To the best of our knowledge, we are the first

  • Concatenating the original sentence as context guides rewriting toward greater semantic similarity than rewriting without contextualization.Figure 1 contrasts the same example with and without this contextualization.Source: Figure 1 · p. 2
  • DP-MLM surpasses previous state-of-the-art methods on many private-rewriting benchmarks, particularly at lower per-token ε budgets.The authors identify proposing BERT-based rewriting and a new contextualization technique as contributions.Source: §1 · p. 2

2 Related Work

Prior local-DP text privatization ranges from word-level perturbation to sentence-level generation and language-model rewriting. These approaches differ in how they handle context and the utility costs or evaluation challenges they entail.

2 Related Work

  • Word-level privatization perturbs tokens, often through calibrated noise on word embeddings, without accounting for grammar or context.These methods use metric DP, which can improve utility but complicates comparative evaluation.Source: §2 · p. 2
  • Sentence-level methods account for grammar and context but offer stricter privacy guarantees at a significant utility cost.Examples add noise to latent representations or use additional public information.Source: §2 · p. 2
  • Recent local-DP language-model methods treat privatization as paraphrasing or rewriting, sampling output tokens under local DP.DP-MLM builds on these methods to improve utility- and privacy-preserving capabilities.Source: §2 · p. 2

3 Foundations

The foundations explain how MLMs use sentence context to predict masked tokens, then define the privacy setting and sampling mechanism used for DP-MLM. The method maps MLM scores to private token selection through the Exponential Mechanism.

3.1 Masked Language Modeling

  • An MLM predicts a masked token using both left and right context, and can use that prediction to replace a target word.Its masked-token probability is conditioned on the other tokens in the sentence.Source: §3.1 · p. 3

3.2 Differential Privacy

  • Differential Privacy bounds how much a randomized mechanism's output distribution can change between neighboring inputs.For this work, any two context sentences are adjacent.Source: §3.2 · p. 3

3.2.1 Temperature Sampling as an Exponential Mechanism

  • The Exponential Mechanism privately selects a token using an MLM scoring function, whose logits can be clipped to bound sensitivity.With bounded logits, temperature sampling realizes the Exponential Mechanism; the stated relation is ε = 2∆u/T.Source: §3.2.1 · p. 3

4 Method

DP-MLM privatizes text by replacing masked tokens using a masked language model conditioned on the original sentence. It applies a per-token privacy mechanism, with sequential composition giving an n-token text a total budget of nε.

4.1 DP Masked Token Prediction

  • The masked-token mechanism clips logits, scales them by temperature T = 2Δu/ε, and samples a replacement token from the resulting softmax probabilities.The input sentence is provided with the target token masked.Source: §4.1 · pp. 3–4

4.2 Rewriting Mechanism

  • DP-MLM applies the token replacement at each position, using the original sentence concatenated with the masked sentence to provide context.Stopword replacement is disabled in the implementation but remains configurable.Source: §4.2 · p. 4

4.3 Privacy Guarantees

  • Theorem 1 establishes ε-LDP for the single-token mechanism, which makes the sampled token difficult to distinguish between adjacent context sentences.The paper describes this as plausible deniability about the source context.Source: §4.3 · p. 4
  • Sequential composition gives a text of n tokens a total privacy budget of nε because the mechanism is called once per token.Source: §4.3 · p. 5

5 Experimental Setup

The experiments assess utility preservation and empirical privacy, comparing DP-MLM with other rewriting methods on GLUE and review datasets. The setup varies privacy budgets and evaluates both rewritten-text utility and adversarial performance.

5 Experimental Setup

  • The evaluation is organized into utility experiments and empirical privacy experiments, with utility testing divided into benchmarking and comparison against two rewriting methods.Source: §5 · p. 5, §5.1 · p. 5

5.1.1 Utility Benchmarking

  • Utility benchmarking measures rewritten-data performance against a non-privatized baseline across nine GLUE tasks and ε values of 10, 25, 50, 100, and 250.A DeBERTa-v3-base model is fine-tuned for one epoch, and results average three runs.Source: §5.1.1 · p. 5

5.1.2 Comparative Utility

  • Comparative utility testing uses DP-Paraphrase and DP-Prompt on COLA, MRPC, RTE, and SST2, reporting task scores alongside BLEU and cosine similarity.Generative outputs are capped at the input length for comparison.Source: §5.1.2 · pp. 5–6

5.2 Empirical Privacy Experiments

  • The privacy evaluation reports utility F1, adversarial Privacy F1, percentage points above majority-class guessing, cosine similarity, and relative gain.Relative gain compares the utility and privacy scores on rewritten data with their non-privatized baselines.Source: Table 3 · p. 7, §5.2 · pp. 7–8
  • Empirical privacy tests use Trustpilot reviews and Yelp reviews, evaluating protection against gender classification or author identification in static and adaptive attacker settings.The Trustpilot sample contains about 36,000 reviews; the Yelp sample contains 2,500 texts from ten authors.Source: §5.2 · p. 7

6 Results

The results section presents utility benchmarking, comparative utility, empirical privacy, and rewriting-speed results. The reported speed differs across the three evaluated mechanisms.

Utility

  • Table 1 reports DP-MLM utility benchmark scores, while Table 2 and Figure 2 present comparative utility results on a GLUE subset.Source: §6 · p. 8

Privacy

  • Table 3 reports empirical privacy results for Trustpilot and Yelp.Source: §6 · p. 8

Efficiency

  • DP-MLM rewrote 1,048,231 tokens in 1,316 minutes (797 tokens/min), versus 1,308 minutes (802 tokens/min) for DP-Paraphrase and 1,961 minutes (535 tokens/min) for DP-Prompt.Source: §6 · p. 8

7 Discussion

DP-MLM leads the utility comparisons in most settings, while privacy results vary by attack setting and ε. The discussion also introduces variable-length rewriting and its associated privacy bounds.

Utility

  • DP-MLM achieves the highest utility score in 14 of 20 comparative settings and consistently outperforms DP-PARAPHRASE and DP-PROMPT across the selected GLUE tasks.The discussion reports these comparisons across four tasks representing all three benchmark task types.Source: §7 · p. 8
  • In 8 of 14 highest-utility cases DP-MLM does not have the highest CS; at ε = 10, it never has the highest CS yet remains competitive.At ε values of 100 or 250, it provides more rewriting while maintaining competitive utility in all cases.Source: §7 · p. 8

Privacy

  • All three methods reduce adversarial advantage, especially at lower privacy budgets; DP-PARAPHRASE is particularly effective but preserves utility less well.On Trustpilot, DP-MLM and DP-PROMPT retain enough utility for reasonably learnable sentiment classification; on Yelp, DP-MLM alone achieves positive PP+ scores.Source: §7 · p. 8
  • DP-MLM reduces adversarial F1 better than DP-PROMPT at higher ε in the static setting, while the reverse holds in the adaptive setting.Relative gains favor DP-MLM in most cases, with better trade-offs at higher ε particularly under static attacks; it has positive gains in all adaptive scenarios.Source: §7 · p. 9

Efficiency

  • DP-MLM performs nearly identically in efficiency to DP-PARAPHRASE, and both greatly outperform DP-PROMPT.The discussion attributes this to the smaller ROBERTA-BASE and GPT-2 models relative to FLAN-T5-BASE, which is nearly double in size.Source: §7 · p. 9

Addressing Limitations

  • The token-replacement mechanism initially cannot produce sentences of a different length; an add/delete variant deletes input tokens probabilistically and inserts mask-predicted tokens.New tokens are generated by inserting a mask into the context sentence and running DP-MLM as usual.Source: §7 · p. 9
  • With addition and deletion, the worst-case guarantee is 2nε-DP, while the average-case guarantee is (A − D)nε-DP.The worst case is when a token is added for every input token.Source: §7 · p. 9

8 Conclusion

DP-MLM uses masked token prediction in encoder-only models to rewrite text contextually. Experiments report improvements over prior state-of-the-art methods on many utility benchmarks and competitive empirical privacy scores, while identifying evaluation and model choices as future research directions.

Conclusion

  • DP-MLM uses masked token prediction in BERT-based encoder-only models to rewrite text contextually, contrasting with prior methods that generate privatized text using decoder-based language models.Simple concatenation guides private sampling of replacement tokens.Source: §8 · p. 9
  • Experiments show improvements over prior state-of-the-art methods on many utility benchmarks and competitive empirical privacy scores, with a reported balance between utility and privacy.The conclusion characterizes this balance as better than that of previous state-of-the-art methods.Source: §8 · p. 9
  • Future work includes encoder-only models, reducing utility loss during privatization, comparing base models, and unifying evaluation strategies for DP text rewriting.The authors call for well-defined methodologies to validate future mechanisms.Source: §8 · p. 9

Limitations

The evaluation uses one representative language model for each mechanism, leaving other BERT-based models outside the study. Privacy protection is evaluated empirically by proxy.

Limitations

  • DP-MLM is represented by ROBERTA-BASE, while DP-PARAPHRASE and DP-PROMPT use GPT-2 and FLAN-T5-BASE, respectively.The implications of evaluating other BERT-based models were outside this work's scope.Source: Limitations · p. 10
  • The authors identify both model selection and evaluation of privacy protection in NLP as limitations.Source: Limitations · p. 10
  • The privacy evaluation follows a predominant literature method but is only a proxy, so privacy protection is claimed empirically and by proxy.Source: Limitations · p. 10

Ethics Statement

The empirical privacy experiments simulate adversaries using existing public datasets that were not originally intended for adversarial gender or author identification.

Ethics Statement

  • The empirical privacy experiments use existing datasets not originally intended for adversarial gender or author identification.Source: Ethics Statement · p. 10
  • The experiments simulate an adversary leveraging publicly accessible information to create an adversarial model.Source: Ethics Statement · p. 10
  • The authors report no harm from the experiments; Yelp uses pseudonymous User IDs rather than PII, further reducing potential harm.Source: Ethics Statement · p. 10

A Implementation Details of DP-MLM

The implementation combines MLM token predictions with a context-similarity score, while fixing α and disabling optional WordNet filtering in the evaluation. The appendix also documents comparison-method replications, additional results, and rewritten examples.

A Implementation Details of DP-MLM

B Implementation of Previous Works

C Additional Results

  • With token addition and deletion, results show higher utility but lower empirical privacy, especially in the adaptive setting.The experiments use ε values 25, 50, and 100, addition probabilities 0.1 and 0.25, and deletion probability 0.05.Source: C Additional Results · p. 12

D Privatization Examples

Loading 2407.00637v1…