PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing large-language models to generate private synthetic text

Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, Andreas Terzis

arXiv:2306.01684v2cs.LGcs.CR

TL;DR

Releasing DP synthetic text can support reuse beyond a single private model, but generating high-quality private synthetic text is difficult and earlier LLM approaches had design and contamination concerns. This paper combines target-focused Prefix-LM training with parameter-efficient private fine-tuning, and reports downstream performance comparable to or better than DP training on real data, with synthetic data also useful for classifier tuning.

  • Problem

    Releasing only a differentially private model does not let others reuse the underlying data for analysis or different models, while generating useful DP synthetic text is difficult.

    Source: §1 · pp. 1–2

  • Method

    The paper privately fine-tunes a pre-trained LLM using Prefix-LM and parameter-efficient methods such as LoRA, then samples DP synthetic data for downstream training and tuning.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 5

  • Results

    Across experiments, models trained on private synthetic data were comparable to or better than models trained on real data with DP; LoRA yielded up to 11 percentage points lift in downstream model performance.

    Source: §1 · p. 2, §5.1 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    DP synthetic data can support downstream classifier training and hyperparameter tuning, with trial rankings correlating up to 87% with real-data ordering even at ϵ = 1.

    Source: §1 · p. 2

    Experiments covered only English datasets; the authors expect extension to multilingual data when public multilingual pre-training data are available.

    Source: §7 · p. 9

Abstract

from arXiv · show

Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, and shared with third parties without sacrificing privacy. However, generating private synthetic data is much harder than training a private model. To improve performance on text data, recent work has utilized public data by starting with a pre-trained generative language model and privately fine-tuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. Further, we demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.

1 Introduction

The paper argues for releasing differentially private synthetic text rather than only a private model, so others can reuse the data for analysis or new models. It addresses prior difficulties with LLM-based synthesis through changes to privacy handling and fine-tuning, reporting strong downstream performance and utility for tuning classifiers.

1 Introduction

  • DP synthetic data could let others reuse sensitive-data-derived datasets, whereas releasing only a private model or its predictions is inadequate for many applications.The paper focuses on central DP and notes that generating synthetic data is harder than privately training a model.Source: §1 · pp. 1–2
  • Earlier LLM-based approaches had mixed success, with poor utility or modified objectives, and could overlook privacy leakage from pre-training on sensitive benchmarks.The paper identifies GPT-2 pre-training on datasets such as IMDB as a potential source of leakage.Source: §1 · p. 2
  • The proposed execution de-duplicates pre-training data, uses long text sequences as the privacy unit, and applies Prefix-LM loss and parameter-efficient tuning.The authors argue that standard private fine-tuning does not yield the desired synthetic-data quality.Source: §1 · p. 2
  • Up to 11 percentage points lift in downstream model performance is reported for LoRA tuning, which the authors find superior to full fine-tuning under DP.Across experiments, models trained on private synthetic data are comparable to or better than downstream models trained on real data with DP.Source: §1 · p. 2
  • Generating more synthetic data can help simpler downstream models, and the data can tune classifiers with up to 87% ranking correlation to real-data trial ordering, even at ϵ = 1.The paper presents synthetic-data-based hyperparameter tuning as an additional use of the generated data.Source: §1 · p. 2

2 Related work

Prior synthetic-text methods often relied on heuristic anonymization or reported utility losses, while failing to address pre-training contamination. This paper highlights parameter-efficient private fine-tuning as an underexplored route to higher-quality DP synthetic data.

2 Related work

  • Many earlier complex-data synthesis approaches used heuristic anonymization without formal privacy guarantees, including a clinical-note study that demonstrated privacy only empirically.That study used heuristic rules to de-identify summaries generated with an LSTM.Source: §2 · pp. 2–3
  • Prior DP-fine-tuned LLM studies had mixed utility: some reported up to 25% downstream accuracy drops on synthetic data, while others reported large classifier-performance drops or modified the training objective.Bommasani et al. proposed but did not evaluate a method; Putta et al. and Mattern et al. [26] proposed additional mechanisms or loss terms.Source: §2 · p. 3
  • Earlier work did not account for overlap between LLM pre-training data and downstream datasets, which may invalidate privacy guarantees or inflate accuracy estimates.The paper reports finding some downstream training and test examples in GPT-2 pre-training data.Source: §2 · p. 3
  • The authors identify private parameter-efficient fine-tuning, especially LoRA, as an unexplored approach that can outperform full fine-tuning for DP synthetic-data quality.They state that prior DP synthetic-data work had not explored parameter-efficient fine-tuning.Source: §2 · p. 3

3 Preliminaries

The preliminaries define the paper’s privacy setting and contrast DP synthetic data with privacy applied only to a particular model. They also introduce the training and language-model terminology used in the paper.

3 Preliminaries

  • The paper uses (ϵ, δ)-DP with neighboring datasets differing by the addition or removal of one record.Smaller ϵ and δ correspond to stronger privacy guarantees.Source: §3 · p. 3
  • DP synthetic data protects the original data at the input level, so any model trained on it is DP, unlike DP training that protects only a particular model.The authors expect downstream models trained on DP synthetic data to perform at most as well as the corresponding DP-trained models on real data, while noting that pre-trained LLMs inject substantial public data.Source: §3 · p. 3
  • DP-SGD clips per-example gradients and adds Gaussian noise to their aggregate, with noise tied to the clipping norm and privacy strength.The same recipe can be used with adaptive optimizers such as Adafactor.Source: §3 · p. 3
  • Pre-training is initial LLM training on a large public dataset, while fine-tuning adapts a pre-trained model to a task by running optimizer epochs on task data.Source: §3 · p. 4

4 Methodology

The methodology privately adapts a pre-trained language model, samples synthetic training and validation data, and trains and evaluates downstream models. It uses target-focused Prefix-LM training and parameter-efficient tuning, then samples at least as much synthetic data as the original training set.

4 Methodology

  • The pipeline privately fine-tunes a public generative LLM, samples synthetic training and validation splits, trains a downstream model on them, and evaluates it on the real test set.The synthetic training data is intended for use by a non-trusted third party.Source: §4 · p. 4, Figure 1 · p. 4

4.1 Using an LLM for data synthesis

  • For synthesis, the model receives a task-and-label prefix and uses the text example as its target under the Prefix-LM objective.The paper uses decoder-only LLMs in its experiments.Source: §4.1 · p. 4
  • Prefix-LM assigns zero loss weight to prefix tokens, avoiding gradients that teach the prompt distribution and are corrupted by DP noise.The authors say this setup is beneficial for private training and supports one model for all class labels.Source: §4.1 · p. 5

4.2 Parameter-efficient fine tuning

  • Because private full fine-tuning can be costly and lose utility, the paper tests prompt tuning and LoRA; prompt tuning updates only a small prompt tensor while freezing the model.The authors report that prompt tuning required a random-uniform initialization and optimizer changes to improve utility and stability.Source: §4.2 · p. 5
  • LoRA freezes pre-trained weights and adds trainable low-rank matrices to dense layers; the authors report it outperforming prompt and full fine-tuning.They prefer tuning both MLP layers and attention blocks and call LoRA the most promising parameter-efficient technique so far.Source: §4.2 · p. 5

4.3 Data sampling

  • Sampling randomly selects a label, forms its task-and-label prefix, and autoregressively generates examples until the target dataset size is reached.The experiments sample at least as much synthetic data as the original training set; more data generally helps downstream performance, with diminishing benefit and compute as a limit.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate private synthetic data for downstream classification, hyperparameter tuning, and synthetic-data quality estimation. Private LoRA-tuned synthetic data performs similarly to or better than direct DP training in the reported comparisons.

5 Experiments

  • Experiments use IMDB, Yelp, and AGNews for binary classification, comparing BERT and CNN models trained on real, DP-on-real, or synthetic data.The generative model's pre-training data is deduplicated against sensitive datasets to prevent overlap; real-data baselines include non-private and directly DP-trained classifiers.Source: §5.1 · p. 6, §5 · p. 6

5.1 Downstream classifier performance

  • Up to 10-11% lift on downstream BERT tasks comes from LoRA-tuned versus fine-tuned synthetic data, while prompt-tuned data gives CNN up to 22% boost.The authors hypothesize that fewer updated parameters reduce noise for LoRA, while prompt tuning's minimal updates can hurt utility, suggesting a parameter-count sweet spot.Source: §5.1 · p. 7
  • Private LoRA-tuned synthetic data matches or exceeds DP-on-real for BERT and outperforms it for CNN, while also supporting data sharing and hyperparameter tuning.The comparison covers IMDB, Yelp, and AGNews; the table caption identifies accuracy for BERT and CNN classifiers.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 Tuning downstream model hyperparameters on synthetic data

  • Synthetic-data validation accuracy shows excellent ranking correlation with real test accuracy for hyperparameter selection; better-fidelity prompt- or LoRA-tuned data also makes BERT performance less sensitive to hyperparameters.The experiments tune weight decay and learning rate, and compare synthetic-validation rankings with real-validation rankings as an upper bound.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 Estimating synthetic data quality

  • Among proxy metrics for synthetic-data quality, test-set perplexity performs very well and is recommended when available because it is easy to compute during fine-tuning.The authors also find that Sentence-T5 embeddings yield higher MAUVE correlation with downstream performance than BERT or other tried embedding models.Source: §5.3 · p. 9, Table 4 · p. 9

6 Conclusion

The conclusion presents LoRA-tuned private synthetic data as an effective alternative to direct DP training for text classification. It also highlights the usefulness of standard Prefix-LM loss and synthetic data for downstream work.

6 Conclusion

  • LoRA-tuning produces downstream performance comparable to or better than direct DP training on real data, and is superior to full fine-tuning for obtaining DP synthetic data.The paper reports up to an 11-percentage-point lift in downstream model performance with LoRA-tuning.Source: §6 · p. 9
  • The standard NLP Prefix-LM loss is well suited to privately fine-tuning language models for synthetic data generation.Source: §6 · p. 9
  • DP synthetic data can be used for feature engineering, hyperparameter tuning, debugging, monitoring, and sharing without privacy-related concerns.Source: §6 · p. 9
  • Mauve is a good proxy for synthetic-data quality, while simpler metrics such as perplexity perform well when available.Source: §6 · p. 9

7 Ethics statement

The authors expect private synthetic data to support safer sharing with specialists and a positive societal impact. Their experiments, however, cover only English datasets.

7 Ethics statement

  • The authors expect private synthetic data sharing to let data owners work with specialists who can create models using their expertise without compromising contributors’ privacy.Source: §7 · p. 9
  • They expect this approach to benefit entities that own private data but lack the knowledge or resources to train predictive models.Source: §7 · p. 9
  • The experiments cover only English datasets; the authors expect multilingual methods to work if public multilingual pre-training data are available.Source: §7 · p. 9

SUPPLEMENTARY MATERIAL

The supplement specifies the scope and assumptions of the privacy guarantees, explains how next-token losses are aggregated for example-level privacy, and reviews prior synthetic-tabular-data approaches.

A Our privacy guarantees

  • The guarantee is central DP under a trusted service provider, covers a single training run but not hyperparameter tuning, and uses example-level privacy for full text sequences up to 512 tokens with add-or-remove adjacency.The accounting uses RDP with Poisson sampling assumed for privacy amplification.Source: A Our privacy guarantees · p. 15
  • The released predictions or synthetic data are protected, while the mechanism’s formal output is the privatized gradients; checkpoints are also protected and may be released publicly.Source: A Our privacy guarantees · p. 15

A.1 Preserving example-level privacy

  • For example-level privacy, each review is split into next-token prediction sub-examples, whose weighted losses are aggregated into one example loss.Prefix-LM normalization assigns zero weight to the task prefix and equal weights to the remaining tokens.Source: §A.1 · p. 15
  • DP-SGD clips each example’s total weighted gradient norm before adding noise calibrated to the desired privacy guarantee.Source: §A.1 · pp. 15–16

B Additional related work

  • Synthetic-tabular methods include marginal-based approaches and generative models; a cited benchmark reports marginal-based methods outperforming GAN-based approaches for downstream ML performance.Earlier work often evaluated statistical query answers, while newer work assessed downstream model utility.Source: B Additional related work · p. 16

C Language model pre-training.

The paper uses decoder-only transformer language models, with an 8B model for final tuning and a 1B model for hyperparameter selection. It deduplicates pre-training data against downstream datasets and describes the pre-training procedure.

C Language model pre-training.

  • The experiments use decoder-only transformers; the 8B model is used for final fine-tuning and prompt tuning, while the 1B model tunes hyperparameters.The paper points to Table 5 for the models’ architecture statistics.Source: C Language model pre-training · p. 16, Table 5 · p. 16
  • The Pile is deduplicated against downstream datasets by removing shared sequences of 50 or more tokens to prevent private data from being learned during pre-training.The paper states that such accidental learning would invalidate its DP guarantees.Source: C Language model pre-training · p. 16
  • Pre-training used 380k steps, batches of 1024 examples, and 1024-token sequences, totaling approximately 400B tokens with next-token prediction.The procedure used T5X, AdaFactor, weight decay of 0.001, and an auxiliary z-loss of 10^-4 log2(Z).Source: C Language model pre-training · p. 17

D Analysis of pre-training data contamination of GPT-2 training data.

The appendix examines contamination between GPT-2 pre-training data and downstream datasets, then reports prompt-tuning and LoRA configuration findings. Its contamination analysis finds evidence of IMDB overlap, while the tuning analyses identify initialization and layer-selection choices associated with better performance.

D Analysis of pre-training data contamination of GPT-2 training data.

  • GPT-2’s listed WebText source domains included 183080 IMDB web-pages and 36188 Yelp web-pages, indicating possible overlap with those datasets.The authors note that the unavailable pre-training data prevents estimating what percentage of either dataset was included.Source: D Analysis of pre-training data… · pp. 17–18
  • A c4/webtextlike subset contained 136 distinct IMDB train and test examples; manual review of a Reddit-linked review supported actual inclusion in WebText.The example’s Reddit post predated WebText’s creation, and the authors infer that some IMDB reviews were used in GPT-2 pre-training.Source: D Analysis of pre-training data… · p. 17

E Details of prompt-tuning.

  • Adafactor with vocabulary-based prompt initialization produced significant instability in the best DP runs, even after clipping-norm and learning-rate tuning.The appendix points to Figure 2 for the best run’s training loss, training accuracy, and validation perplexity.Source: E Details of prompt-tuning · p. 18, Figure 2 · p. 18
  • Random-uniform prompt initialization improved downstream performance by up to 30% for CNN and 10% for BERT; the authors hypothesize noise makes recovery from poor initialization difficult.The tested range was [−0.01, 0.01].Source: E Details of prompt-tuning · p. 18, Table 6 · p. 18
  • Adam and Momentum made prompt-tuning runs more stable, but changing optimizers did not improve downstream performance; the authors ultimately used Adam with a fixed learning rate.When tuned, Adam with fixed learning rate, Adam with cosine decay, and Momentum with cosine decay performed similarly.Source: E Details of prompt-tuning · p. 18

F Details of LoRA

  • Full-LoRA, which adapts attention and MLP layers, was best in most cases, though MLP-LoRA was slightly better in some cases.The reported main results use Full-LoRA rank 8 on AGNews, rank 1 on Yelp, and MLP-LoRA rank 32 on IMDB.Source: F Details of LoRA · p. 19
  • LoRA rank initially increased downstream accuracy, but accuracy eventually decreased as rank rose further.The authors recommend Full-LoRA rank 8 as a default while encouraging parameter tuning when resources allow.Source: F Details of LoRA · p. 19

G Details of synthetic data sampling.

The sampling appendix describes temperature, top-k, and number of decodes as generation controls. The default settings generally provide sufficient diversity, with higher temperature or more decodes suggested when additional diversity is needed.

G Details of synthetic data sampling.

  • Temperature scales logits before softmax, top-k restricts sampling to the most likely tokens, and numdecodes selects the highest-likelihood sample among repeated decodes.The study used temperature 1, top-k ∞, and one decode as defaults.Source: G Details of synthetic data sampling · p. 19
  • Higher temperature can increase generated-data diversity but may reduce quality, while top-k limits choices and numdecodes repeats decoding to select a candidate.The authors leave top-k unmodified and use one decode by default.Source: G Details of synthetic data sampling · p. 19
  • Default sampling settings provided enough diversity for the studied datasets; increasing temperature or numdecodes is suggested when more diversity is needed.The stated default temperature is 1, top-k is ∞, and numdecodes is 1.Source: G Details of synthetic data sampling · p. 19

H Details of datasets for downstream tasks

The appendix details the datasets, downstream models, and ablations used to assess synthetic-data utility. Its ablations report effects of model size, training choices, sampling settings, and loss normalization.

H Details of datasets for downstream tasks.

  • IMDB and Yelp were treated as binary sentiment tasks, while AGNews classified news titles into four topics; validation data came from the final 10% of each training set.The original training set was deterministically split into 90% training and 10% validation.Source: H Details of datasets for downstream… · pp. 19–20

I Architecture and training of downstream models.

  • Downstream evaluations used fine-tuned BERT classifiers and an un-pretrained shallow CNN, trained with Adam or DP-Adam and selected by validation accuracy.The BERT encoder was trained without frozen layers; the CNN used randomly initialized embeddings.Source: I Architecture and training of… · p. 20

J Ablations

  • The ablations report that standard DP-training techniques apply to full and parameter-efficient LLM tuning, and Prefix-LM usually performs better than Full LM in the DP setting.Loss normalization was also reported to ease hyperparameter tuning, especially for the clipping norm, and to help the LLM produce outputs of varying lengths.Source: J Ablations · p. 21

J.1 Ablation: pre-training dataset and model size

  • Larger LLMs yielded better downstream performance for both BERT and CNN, while the limited C4-versus-The-Pile study found similar performance.The authors settled on The Pile because it was easier to download and use.Source: §J.1 · p. 21, Table 8 · p. 21

J.2 Ablation: hyperparmeter tuning for DP-training of LLM

  • Downstream performance was sensitive to the LLM learning rate; on IMDB, longer training with larger batches was typically better under DP.The IMDB prompt-tuning sweep used privacy level ϵ = 1.Source: §J.2 · p. 21, Table 9 · p. 21

J.3 Ablation: sampling parameters.

  • Sampling-parameter tuning brought only modest gains over defaults; slightly higher temperatures appeared helpful when paired with tighter top-k or more decodes.The reported higher temperatures were 1.4 for IMDB and 1.2 for Yelp; additional decodes were costly and judged likely not worth that cost.Source: §J.3 · p. 22

J.4 Ablation: loss of LLM.

  • 0.43 accuracy resulted from loss normalization with clipping norm 1 on Yelp, compared with approximately 0.29 accuracy without normalization at clipping norm approximately 2000.The authors say normalization’s benefit diminishes when a full clipping-norm and learning-rate sweep is feasible.Source: §J.4 · p. 22

J.5 Ablation: LoRA parameters.

The LoRA ablation finds that the strongest layer choice differs by dataset, while ranks above 32 add little performance and require tuning more parameters.

J.5 Ablation: LoRA parameters.

  • Full-LoRA performs better than Attention-LoRA and MLP-LoRA in most IMDB cases, while MLP-LoRA appears better overall on AGNews.Full-LoRA is recommended as a reasonable default when layer choices cannot be tuned beforehand.Source: §J.5 · pp. 22–23
  • Performance is typically best at ranks 8–32; ranks above 32 bring little or no gain while requiring more parameter tuning.Source: §J.5 · p. 23

K Evaluating synthetic data quality: Mauve robustness.

The appendix examines MAUVE robustness and lower-cost proxy measures, then compares synthetic and original text distributions, implementation costs, and example data.

K Evaluating synthetic data quality: Mauve robustness.

L Evaluating synthetic data quality: Proxy metric experiments.

  • Proxy metrics offer cheaper estimates for tuning generator and sampling hyperparameters, with rank correlation most relevant to selecting high-performing settings.Figure 4 plots metric estimates against true downstream-classifier performance; metric values are rescaled and their absolute values are uninformative.Source: L Evaluating synthetic data quality:… · p. 25, Figure 4 · p. 26

L.1 Synthetic data text length comparison

  • Text-length distributions predict downstream performance, though less strongly than other measures; synthetic samples are truncated at 512 tokens while 17% of IMDB and 7% of Yelp originals are longer.Source: §L.1 · p. 25, Figure 5 · p. 27, Figure 6 · p. 30

L.2 Synthetic data bigram frequency comparison

  • Bigram distributions show smaller gaps with less noise; full fine-tuning has the largest gap, prompt tuning follows, and LoRA is most aligned with the original.Source: §L.2 · p. 25, Figure 7 · p. 31

M Implementation details and estimates of the required compute

  • Pretraining the 8B model took about eight days; private fine-tuning took 20–80 hours, prompt tuning 1.5–20 hours, and LoRA about twice as long as prompt tuning.LoRA runs ranged from three hours to two days; downstream sweeps used about 28 settings and were repeated three times for error bars.Source: M Implementation details and estimates… · pp. 25–27

N Examples of generated and real data

Related papers

Loading 2306.01684v2…