PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe

Xiang Yue, Huseyin A. Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, Robert Sim

arXiv:2210.14348v3cs.CLcs.CR

TL;DR

Privacy attacks and prior low-quality private synthesis motivate a better way to release useful text without exposing training examples. The paper fine-tunes pretrained generative language models with differential privacy and uses them to generate controlled synthetic text. Across benchmark and customer data, the approach achieves useful, competitive utility while providing formal privacy protection, though strong privacy can omit rare patterns and hurt small classes.

  • Problem

    Existing models can reveal or reconstruct sensitive training data, while previous privacy-preserving synthetic-text approaches often produced low-quality data.

  • Method

    The method fine-tunes pretrained generative language models with differential privacy and generates synthetic text using control codes.

  • Results

    DP-generated synthetic text supports downstream classifiers within 2-4% accuracy of classifiers trained on original reviews and shows distributional similarity to the original data.

  • Takeaways & Limitations

    A small amount of DP synthetic data can support downstream models without additional privacy loss, while the method remains feasible on private customer feedback.

  • Takeaways & Limitations

    Strong DP limits learning the tail of the training distribution, making rare patterns and valuable outlier information unlikely to appear in synthetic text.

Abstract

from arXiv · show

Privacy concerns have attracted increasing attention in data-driven products due to the tendency of machine learning models to memorize sensitive training data. Generating synthetic versions of such data with a formal privacy guarantee, such as differential privacy (DP), provides a promising path to mitigating these privacy concerns, but previous approaches in this direction have typically failed to produce synthetic data of high quality. In this work, we show that a simple and practical recipe in the text domain is effective: simply fine-tuning a pretrained generative language model with DP enables the model to generate useful synthetic text with strong privacy protection. Through extensive empirical analyses on both benchmark and private customer data, we demonstrate that our method produces synthetic text that is competitive in terms of utility with its non-private counterpart, meanwhile providing strong protection against potential privacy leakages.

1 Introduction

The paper studies DP synthetic text generation as an alternative to task-specific private learning, using DP fine-tuning of pretrained language models. Experiments indicate useful downstream performance, fixed privacy loss for unlimited post-processed models, distributional similarity, and a length-truncation effect.

  • Motivation: The method targets synthetic text generation because DP generative models capture distributional properties and support multiple downstream tasks through post-processing.The paper also identifies easier dataset debugging and freer sharing as advantages of synthetic data.
  • Contributions: DP fine-tuning of pretrained language models produces synthetic text whose downstream classifiers are only 2-4% less accurate than classifiers trained on original reviews.The example uses GPT-2 Large fine-tuned with ε = 4.
  • Contributions: Synthetic data enables downstream models without additional privacy loss, whereas repeatedly training private models on original data accumulates privacy loss.The paper reports that a small amount of DP synthetic data can match models trained directly on the entire original dataset with DP.
  • Findings: Distributional evaluation finds that synthetic text resembles the original distribution, while DP-trained models generally generate shorter completions than non-DP models.The shorter-completion pattern is described as a novel phenomenon of independent interest.
  • Evaluation: The study stress-tests privacy leakage with canaries and evaluates feasibility on an industrial-scale private customer-feedback dataset.The customer-feedback study uses one million collected feedback examples.

2 Background

Differential privacy provides a formal bound on how much one sample can affect an algorithm’s output. Its post-processing robustness and DP-SGD implementation are central to privately training and using neural models.

  • Differential privacy: DP makes outputs statistically similar whether any individual sample is included, with ε bounding impact and δ bounding the probability that the guarantee fails.The definition applies to neighboring datasets differing in one sample.
  • Differential privacy: Choosing ε and δ requires balancing privacy and utility for the specific use case.The paper notes that these parameters are selected with the acceptable trade-off in mind.
  • Differential privacy: Post-processing robustness allows arbitrary deterministic or randomized functions of a DP output without additional privacy loss.This property later permits sampling and downstream use of a DP-trained language model.
  • DP-SGD: DP-SGD clips per-sample gradients and adds Gaussian noise to their batch sum, obfuscating individual contributions during neural-network training.The passage describes the modified stochastic-gradient procedure used for deep learning.

3 Method

The method fine-tunes a pretrained generative language model with DP-SGD, then samples synthetic text from it. Control codes condition generation on categories, while post-processing preserves the privacy guarantee.

  • 3.1 Problem Statement: The goal is a randomized mapping from a token-sequence database to a synthetic database that satisfies DP while retaining downstream utility and distributional properties.Utility is defined by comparable performance on relevant tasks and preservation of the original data’s properties.
  • 3.2 Synthetic Text Generation with DP: The conventional baseline trains an autoregressive language model on the original dataset and samples sequences using mechanisms such as beam search or nucleus sampling.The proposed method changes the training stage by applying DP-SGD to pretrained models.
  • 3.2 Synthetic Text Generation with DP: DP-SGD fine-tunes a pretrained generative language model, after which sampling synthetic text incurs no additional privacy loss.This follows from DP’s post-processing property.
  • 3.2 Synthetic Text Generation with DP: Control codes condition autoregressive generation on styles, domains, sentiments, or categories, enabling targeted synthetic samples.The model sequentially samples tokens conditioned on the control code and previously generated tokens.
  • 3.2 Synthetic Text Generation with DP: The implementation prepends each training sample with its categories so generation can preserve the original categorical distribution.The paper notes that this categorical distribution may itself be private and is not privatized here.

4 Analyses on a Public Review Dataset

Experiments on Yelp reviews evaluate DP fine-tuned language models for synthetic text generation, downstream utility, distributional similarity, and privacy leakage. Larger models generally improve utility and similarity, while DP suppresses canary leakage but introduces shorter generations and weaker tail coverage.

  • 4.2 Downstream Tasks on Synthetic Data: GPT2-Large with DP produces downstream models comparable to those from non-private synthetic data and original-data training across review-rating and business-category classification.The DP setting uses ϵ = 4, and larger pretrained models reduce the performance gap between private and non-private generations.
  • 4.3 Synthetic Data Generation with DP v.s. Downstream Task Modeling with DP: Using DP for generation matches direct DP task training at ϵ = 4 while allowing additional downstream models without accumulating privacy loss.The synthetic-data approach fixes the privacy loss at generation, whereas separately training task models on the original private data accumulates loss.
  • 4.3 Synthetic Data Generation with DP v.s. Downstream Task Modeling with DP: A 100K DP-generated synthetic dataset produces superior downstream models to direct DP training on the original 1.9M examples of the same size.This comparison is reported for synthetic data generated by a DP-trained model.
  • 4.4 Similarity between Synth. and Real Data: Distributional quality improves with larger private fine-tuned models; GPT2-Large with DP matches GPT2 without privacy on F1, but gaps remain for FID and MAUVE.Topic distributions are also similar to the original data, with similarity improving as model size increases.
  • 4.4 Similarity between Synth. and Real Data: Synthetic generations are generally shorter than original text, with the shortening more pronounced under DP and especially for smaller models.Larger models mitigate but do not fully resolve the length truncation effect.
  • 4.5 Learning Dynamics with DP: DP prevents injected canaries from appearing in synthetic generations even at repetition rate 100, whereas 4 of 5 appear verbatim without DP.Across repetition levels, DP canaries also avoid low perplexity ranks and are not distinguishable among similar candidates.

5 Results on Private Customer Feedback

On private Microsoft customer feedback, DP-trained GPT2-Large generated synthetic data that supported comparable downstream classification performance, while dataset size and class representation remained important constraints.

  • Background: The private customer-feedback experiment was designed to assess whether the method could safeguard both utility and privacy in a practical scenario.Customer feedback can contain sensitive details such as job titles, company names, or locations.
  • Dataset: The study used 1M customer-feedback records and conditional prompts combining three attributes with their selected values.Prompts followed the form “A1: VA1 | A2: VA2 | A3: VA3”, and GPT2-Large was used for generation.
  • Downstream Task Performance: Comparable downstream classification performance was achieved by models trained on synthetic data from GPT2-Large with DP (ϵ = 4), despite the private customer-feedback setting.The evaluation covered three attributes, A1, A2, and A3, with 5, 45, and 5 classes respectively.
  • Downstream Task Performance: Dataset size and class representation constrained performance, particularly for A2, which had many more classes than the other attributes.The supplied passage attributes the result primarily to the dataset being roughly half the size used earlier and to A2 having a larger class set.

6 Related Work

Related work has studied DP synthetic generation for tabular and image data and privacy-preserving NLP tasks, but prior work had not addressed synthetic text generation through DP language-model fine-tuning.

  • Synthetic Data Generation with DP: Prior DP synthetic-data research primarily addressed tabular and image generation using Bayesian networks, marginal preservation, or DP-trained generative adversarial networks.These approaches targeted the privacy-utility trade-off in non-text data settings.
  • Synthetic Data Generation with DP: One-to-one token- or sentence-level perturbation methods do not satisfy the paper’s requirement that generation be statistically similar whether a sample appears in the dataset or not.The paper contrasts perturbing individual samples with learning a distribution-level generative model.
  • DP Fine-tuning of Language Models: DP fine-tuning had been applied to several NLP tasks, including classification, table-to-text generation, dialogue generation, and semantic parsing, but not synthetic text generation.The paper positions its work as an empirical study of this previously unstudied application.

7 Conclusion

The paper presents pretrained language models combined with differential privacy as a simple recipe for generating high-quality synthetic text while mitigating privacy risks.

  • Conclusion: Pretrained language models provide high-quality synthetic text, while differential privacy supplies formal guarantees limiting any single training example’s influence.The paper evaluates both the utility and privacy risks of the resulting synthetic data.
  • Conclusion: The reported experiments demonstrate high-utility synthetic text with strong privacy protection.

8 Limitations

The method captures broad text statistics but can miss rare patterns and disproportionately affect small classes; its privacy-utility trade-off remains incompletely explored.

  • Limitations: Differential privacy can limit learning of rare patterns, so synthetic text may omit valuable information from outliers in the training distribution.The method does not perfectly replicate all details of the original text.
  • Limitations: Tight privacy guarantees most negatively affect small-size classes, creating a risk for minority populations in conditional private generation.The authors identify mitigation for this effect as future work.
  • Limitations: The study used privacy parameters ϵ = 4 and δ = 1/(N · log N), leaving a more extensive privacy-utility analysis for future work.
  • Limitations: Strong empirical privacy from DP generally depends on individual samples having low or no correlation, so deployments may also require deduplication, redaction, and meaningful example boundaries.The authors connect this limitation to canary extraction experiments involving information repeated across training instances.

9 Ethics Statement

The paper frames synthetic text generation as preserving original statistical properties while protecting individual privacy, using differential privacy and encouraging responsible use.

  • The method incorporates differential privacy as the paper’s privacy-mitigation technique for synthetic generations.
  • The authors acknowledge potential misuse and encourage researchers and practitioners to consider ethical implications and follow data-privacy best practices.

A Implementation Details and Hyperparameters

The appendix documents datasets, training settings, synthetic-text evaluation, and privacy-leakage experiments used to assess DP language-model generation.

  • Data preparation: The study samples 10 frequent business categories and removes reviews without ratings.The categories include Restaurants, Bars, Shopping, Event Planning & Services, Beauty & Spas, Arts & Entertainment, Hotels & Travel, Health & Medical, Grocery, and Home & Garden.
  • Model training: Models trained without DP and with DP use separate hyperparameter settings documented in Tables 7 and 8.The cited implementation uses 16 Tesla V100 GPUs for Yelp and 2 Tesla A100 GPUs for internal feedback data in both settings.
  • Downstream classification: Roberta-base classifiers use batch size 64, learning rate 3e-5, and 5 epochs for downstream text-classification tasks.
  • Synthetic-text evaluation: Synthetic text is evaluated with F1, FID, and MAUVE using five sentence-transformer models whose metric values are averaged.F1 measures sample quality and coverage using nearest-neighbor surroundings; FID compares embedding means and covariances, while MAUVE compares synthetic and original distributions.
  • Privacy evaluation: Privacy leakage is tested with five canary types and 10,000 similar named-entity candidates per canary type.Candidate entities come from web-based lists or randomly generated patterns, and Figure 5 compares perplexity distributions from GPT2 models trained with and without DP.
  • Synthetic-data generation: Canary reviews are synthesized with text-davinci-003 using a Van Gogh restaurant-review prompt, varied sampling hyperparameters, and duplicate filtering.

E Sequence Length Distribution of the Original and Synthetic Data Generated with and without DP

This section compares sequence-length distributions for original customer feedback and synthetic data generated with and without differential privacy.

  • The figure plots sequence-length distributions for original customer feedback and synthetic data generated with and without DP.

F Sampled Synthetic Data

The section presents randomly sampled DP-generated reviews across three GPT-2 model sizes, alongside distributional and privacy-oriented visualizations.

  • Fifteen synthetic examples are randomly sampled from GPT2, GPT2-Medium, and GPT2-Large generations.The examples are provided in Tables 10, 11, and 12, respectively.
  • Figure 4 compares sequence-length distributions for synthetic data generated with and without DP and for original customer feedback.
  • Figure 5 compares perplexity distributions of injected private-information canaries among similar candidates under GPT2 models trained with and without DP.The caption states that even single-occurrence private information can rank at the top in a non-private model, unlike models trained with DP.
Loading 2210.14348v3…