PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring The Empirical Privacy Landscape of Synthetic Data

Zihao Zhao, Avi Schwarzschild, Casey Meehan, Katherine Lee, Zhili Feng, Dylan Sam

arXiv:2610.22659v1cs.CRcs.CL

TL;DR

Training language models on sensitive data raises a capability–privacy trade-off, and prior studies have not systematically compared data- and training-level methods on both dimensions. This paper evaluates those approaches on synthetic medical records with canaries and finds privacy-aware rewriting a promising balance of capability and empirical privacy.

  • Problem

    Prior studies typically evaluate individual privacy-preserving methods or focus on privacy or utility alone, leaving data- and training-level methods without a controlled comparison on both capability and leakage.

    Source: §1 · pp. 1–2

  • Method

    The paper compares data- and training-level interventions by finetuning models on synthetic medical records containing canaries and evaluating privacy leakage and capability.

    Source: §1 · p. 2

  • Results

    Privacy-aware rewriting achieves the most balanced overall trade-off; stronger rewrite models generally improve privacy, while capability and gains from additional reasoning effort are not monotonic.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Privacy-aware rewriting is a promising direction for balancing privacy and capability, and data-level privacy judgments alone cannot replace downstream evaluation of finetuned models.

    Source: §1 · p. 2, §5 · p. 10

    Rewriting requires additional computation because each training record must pass through a generative model before finetuning.

    Source: §4 · p. 6

Abstract

from arXiv · show

Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-off between capabilities and empirical privacy compared to other training-level methods, and our evaluation framework supports comparisons of future models and methods.

1 INTRODUCTION

Training language models on sensitive records can improve high-stakes capabilities but risks exposing private information. This study compares data- and training-level protections in a controlled setting and finds privacy-aware rewriting promising for balancing capability and privacy.

1 INTRODUCTION

  • Data-level methods modify or replace training records, whereas training-level methods modify optimization; DP-SGD limits individual examples’ influence and provides formal privacy guarantees.Synthetic data generation aims to preserve useful semantics while removing sensitive details.Source: §1 · p. 1
  • Existing studies usually assess individual privacy-preserving methods or focus on privacy or utility alone, leaving data- and training-level approaches without a controlled comparison on both dimensions.Prior work also describes a trade-off: stronger privacy often comes with reduced utility, while preserving capabilities may increase leakage risk.Source: §1 · pp. 1–2
  • The paper compares finetuning methods on synthetic medical records and evaluates each for privacy leakage and model capability.The controlled setup aims to attribute leakage to data introduced during the experiments rather than prior pretraining exposure.Source: §1 · p. 2
  • Rewriting provides a promising capability–privacy balance, but its strategy matters; privacy-aware rewriting improves protection while more capable rewrite models generally do better.Additional test-time compute does not monotonically improve the trade-off, and held-out fit does not improve monotonically.Source: §1 · p. 2

2 RELATED WORK

Related work frames privacy leakage around training-data memorization and evaluates defenses at both the data and training stages. It also distinguishes verbatim extraction from broader recovery and inference risks.

2 RELATED WORK

  • Canaries and extraction attacks probe whether models reproduce training sequences, while finetuning data can also be memorized.Susceptibility varies across full-model, output-head, and adapter-based finetuning.Source: §2 · pp. 2–3
  • Memorization can increase with model capacity and example duplication, and approximate copies can induce it; repeated exposure and training position also affect risk.Examples become more likely to be regenerated as training frequency rises, and deduplication can reduce extraction risk.Source: §2 · p. 3
  • Privacy risk extends beyond exact reproduction: information may remain recoverable, and verbatim memorization does not necessarily predict recovery through natural-language questions.The paper therefore evaluates prefix-based completion and question-based recall separately.Source: §2 · p. 3
  • De-identification and identifier surrogates can reduce leakage without eliminating it, while contextual clues may still reveal personal attributes.Context-sensitive anonymization replaces details according to re-identification risk; this paper contrasts that approach with PII scrubbing.Source: §2 · p. 3
  • Training-level defenses include DP-SGD and Goldfish, but rewritten text is not inherently private and the paper’s one-to-one rewriting provides no formal differential-privacy guarantee.The study compares preprocessing and training defenses through leakage and capability in the resulting finetuned models.Source: §2 · p. 3

3 EXPERIMENTAL SETUP

The experiments use synthetic medical records and planted canaries to compare finetuning and privacy-preserving methods under shared training and evaluation conditions. Privacy and capability are measured with separate perplexity metrics.

3.1 DATASET AND CANARY CONSTRUCTION

  • The study uses 5,000 Synthea records for training and a disjoint held-out set, then adds 100 canary records with fabricated identities and planted medication names.Canaries are inserted at multiplicities ×1, ×4, ×16, ×32, and ×64, yielding 7,340 training instances.Source: §3.1 · pp. 3–4

3.2 BASELINES AND PRIVACY-PRESERVING METHODS

  • Six settings share a pretrained model and corpus but vary training objective or preprocessing, including direct finetuning, PII scrubbing, DP-SGD, Goldfish, rewriting, and privacy-aware rewriting.Privacy-aware rewriting instructs the model to replace rare, identifying details with plausible fictional alternatives while retaining common clinical information.Source: §3.2 · p. 4

3.3 MODEL AND TRAINING CONFIGURATION

  • The study evaluates Qwen3.5-4B-Base and gpt-oss-20b using LoRA adapters for two epochs, with the same fixed-order records and random seed.The target models differ in family and scale; gpt-oss uses its native chat format while Qwen uses causal language modeling.Source: §3.3 · p. 4

3.4 PRIVACY AND CAPABILITY METRICS

  • Privacy leakage is measured with verbatim and semantic perplexity, while capability is measured with held-out record and medication perplexity.Lower privacy-probe perplexity means more leakage; lower capability-metric perplexity means better capability.Source: §3.4 · pp. 4–5

4 RESULTS

Across the evaluated methods, stronger privacy protection generally trades off against capability, while rewriting-based methods occupy a more balanced region. Privacy-aware rewriting improves both ranks over standard rewriting, but its outcomes depend on the rewriter and other choices.

4 RESULTS

  • Direct finetuning achieves the highest capability improvements but relatively poor privacy, while DP-SGD has a relatively high privacy rank and lower capability than most trained methods.PII scrubbing ranks poorly on privacy, and Goldfish retains high capability but performs poorly in privacy.Source: §4 · pp. 5–6
  • Rewriting-based methods offer relatively balanced trade-offs, and privacy-aware rewriting improves both privacy and capability ranks over standard rewriting for both target models.Rewriting requires additional computation to process each record; GPT-5.5 has higher privacy but lower capability ranks than Qwen3.5-4B.Source: §4 · p. 6, Figure 3 · p. 7

4.1 CANARY DUPLICATION

  • At higher canary-duplication levels, DP-SGD and GPT-5.5 privacy-aware rewriting maintain higher perplexity, while PII scrubbing tracks direct finetuning.Because canary groups differ in intrinsic difficulty, trained conditions are interpreted relative to the no-finetuning reference at the same duplication level.Source: §4.1 · p. 6, Figure 2 · p. 7

4.2 REWRITING

  • Adding a privacy-aware instruction improves privacy rank for every rewriter comparison shown, and larger rewrite models generally achieve higher privacy ranks.Capability improves in most instruction comparisons, but later GPT-5 variants’ stronger privacy protection does not ensure monotonically improving capability.Source: §4.2 · p. 6, Figure 3 · p. 7, Figure 4 · p. 8
  • Increasing reasoning effort does not monotonically improve privacy or capability ranks, although reasoning generally yields a more favorable trade-off than no reasoning.The best reasoning setting depends on the target model.Source: §4.2 · p. 7, Figure 4 · p. 8
  • Adding summarization generally yields similar or lower capability than privacy-aware rewriting alone, while its effect on privacy leakage varies across models.The evaluated pipeline summarizes each record and then expands the summary into a new record.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
  • Five independent rewrites trained for one epoch each and one rewrite trained for five epochs have largely overlapping metric profiles, with no clear advantage for independent rewrites.The single-rewrite, five-epoch condition performs slightly better overall despite the additional generation cost of independent rewrites.Source: §4.2 · p. 8, Figure 6 · p. 9

4.3 PRIVACY AND CAPABILITY METRICS FOR DATA

  • LLM-judged privacy strongly correlates with verbatim perplexity (Spearman’s ρ = 0.89), but judged quality correlates negatively with measured capability (ρ = −0.64).Judging may be a secondary privacy-risk signal, but it cannot replace direct capability evaluation.Source: §4.3 · pp. 9–10, Figure 7 · p. 9

5 CONCLUSION

The controlled evaluation identifies privacy-aware rewriting as the most balanced overall trade-off among evaluated methods. Its capability metrics differ little from direct finetuning, while privacy metrics indicate substantially reduced leakage; stronger rewrite models generally improve privacy protection.

5 CONCLUSION

  • Privacy-aware rewriting achieves the most balanced overall trade-off among evaluated methods, though capability and additional-reasoning benefits are not monotonic.Source: §5 · p. 10

A.3 SCALING REWRITE MODEL CAPABILITIES

  • Stronger rewrite models generally provide better privacy protection, although capability does not consistently improve.Source: §5 · p. 10

A.1 PRIVACY–CAPABILITY TRADE-OFFS

  • Compared with direct finetuning, privacy-aware rewriting shows only small capability-metric differences but substantially higher verbatim and semantic perplexity, indicating reduced privacy leakage.Source: Figure 9 · p. 14, §A.1 · p. 14, Figure 10 · p. 15

A.2 CANARY DUPLICATION

  • Figure 11 plots semantic perplexity across canary duplication factors for Qwen3.5-4B-Base and gpt-oss-20b, with 20 canaries per group.Higher values indicate less privacy leakage.Source: Figure 11 · p. 15

A.4 SCALING REWRITE TEST-TIME COMPUTE

Additional reasoning during rewriting does not consistently improve privacy or capability, while reasoning-enabled settings generally provide better privacy than no reasoning. Summary-based rewriting has model-dependent effects and remains a practical alternative when finetuning compute is limited.

A.4 SCALING REWRITE TEST-TIME COMPUTE

  • Increasing rewriting reasoning effort does not consistently improve privacy or capability, although reasoning-enabled settings generally provide better privacy than the no-reasoning setting.Source: §A.4 · p. 17

A.5 SUMMARY-BASED REWRITING

  • The effects of summary-based rewriting remain model-dependent, and its effect on verbatim perplexity varies across target models.Source: §A.5 · p. 17
  • Privacy-aware summary-based rewriting remains a practical alternative to rewriting without summarization when finetuning compute is limited.Source: §A.5 · p. 17

A.6 INDEPENDENT REWRITES VERSUS REPEATED TRAINING

For rewriting, repeated training of one rewrite per record outperforms training on five independent rewrites. Judged privacy tracks measured privacy, whereas judged data quality does not reliably predict model capability.

A.6 INDEPENDENT REWRITES VERSUS REPEATED TRAINING

  • Training on one rewrite per record for five epochs outperforms training on five independent rewrites per record for one epoch.Source: §A.6 · p. 17

A.7 PRIVACY AND CAPABILITY METRICS FOR DATA

  • LLM-judged privacy tracks measured privacy, as judged privacy aligns with semantic perplexity.Source: §A.7 · p. 17
  • Judged data quality does not reliably predict model capability, as assessed by medication perplexity.Source: §A.7 · p. 17

A.8 CORRELATION OF PRIVACY METRICS

Verbatim and semantic perplexity are correlated, but their relationship varies substantially across thresholds. The reported patterns support using both metrics, particularly when verbatim perplexity is low.

A.8 CORRELATION OF PRIVACY METRICS

  • For Qwen3.5-4B-Base, verbatim and semantic perplexity correlate at 0.84 overall, 0.39 below verbatim perplexity 100, and 0.98 above that threshold.Source: §A.8 · pp. 17–18
  • For gpt-oss-20b, verbatim and semantic perplexity correlate at 0.85 overall, 0.05 below verbatim perplexity 100, and 0.68 above it.Source: §A.8 · p. 18
  • The differing correlations support reporting both privacy metrics because they provide complementary measurements, particularly when verbatim perplexity is low.Source: §A.8 · p. 18

A.9 CORRELATION OF CAPABILITY METRICS

The appendix compares capability metrics and examines how often target models reproduce coined medications as canary duplication increases. Capability metrics capture different aspects of model performance, while privacy-aware rewriting and DP-SGD keep typical canary-generation probabilities very low.

A.9 CORRELATION OF CAPABILITY METRICS

  • For Qwen3.5-4B-Base, held-out record perplexity correlates 0.60 with medication perplexity overall, 0.52 below 2.5, and 0.44 above 2.5.For gpt-oss-20b, the corresponding correlations are 0.81 overall, 0.90 below 2.5, and 0.75 above it; the authors say both metrics should be reported because they capture different aspects of capability.Source: §A.9 · pp. 18–19, Figure 17 · p. 19, Figure 18 · p. 19

A.10 PROBABILITY OF GENERATING THE COINED MEDICATION

  • The probability measure is the product of a model’s probabilities for the coined medication’s tokens, representing the chance one sampled continuation reproduces it.It is reported against the number of times each canary appeared in training.Source: §A.10 · pp. 19–20
  • For GPT-5.5 rewriting, coined-medication generation probabilities range from 10^-9 to 10^-3, while DP-SGD remains below 10^-11 at every duplication level.Under privacy-aware GPT-5.5 rewriting, typical canaries also remain at the floor; the ×16 and ×64 increases each come from one leaked canary.Source: §A.10 · p. 20
  • After direct finetuning, Qwen3.5-4B-Base generates a medication seen once 16% of the time and one seen 64 times 99% of the time; gpt-oss-20b reaches 69% at ×64.For gpt-oss-20b, a medication seen once is generated about once in 37,000 samples; PII scrubbing rises from 7% to 94%, while Goldfish levels off at 30–40%.Source: §A.10 · p. 20

A.11 LIKELIHOOD-BASED EXPOSURE OF CANARY MEDICATIONS

This section measures how strongly a target model favors each planted medication over unplanted alternatives. Direct finetuning and PII scrubbing show high exposure, whereas rewriting and DP-SGD reduce it.

A.11 LIKELIHOOD-BASED EXPOSURE OF CANARY MEDICATIONS

  • Exposure ranks the planted medication among 6,301 candidates using summed token log probabilities conditioned on the original record prefix.The reported metric is log2(6,301/rank), averaged across 100 canaries, alongside the proportion ranked first.Source: §A.11 · p. 20
  • Direct finetuning reaches mean exposure of 12.22 bits for Qwen3.5-4B-Base and 10.98 bits for gpt-oss-20b, with the planted medication ranked first for 86% and 73% of canaries.PII scrubbing produces similar exposure, while Goldfish leaves the planted medication highest-ranked for 53% and 37% of canaries.Source: §A.11 · p. 20
  • Privacy-aware GPT-5.5 rewriting yields mean exposures of 2.21 and 1.71 bits, while DP-SGD achieves 1.74 and 1.56 bits with no rank-1 canaries observed.DP-SGD averages are close to the corresponding no-finetuning values of 1.49 and 1.39 bits.Source: §A.11 · pp. 20–21
  • Below verbatim perplexity 100, agreement between verbatim and semantic privacy perplexity drops to ρ = 0.39 and 0.05 in the two zoomed views.The figure compares the two privacy metrics across nine method configurations, with rewriting and summary variants shown as gray points.Source: Figure 20 · p. 21

B ADDITIONAL EXPERIMENT DETAILS

The appendix provides prompts for summary-based rewriting.

B.1 SUMMARY-BASED REWRITING

  • The summary-based rewriting prompts closely follow those used for rewriting and are provided in Table 4.Source: §B.1 · p. 21

C CASE STUDY

The case study traces how a representative canary record is handled across methods and what reaches training. The examples distinguish full-text transformations, summary-based methods, and methods that leave the record unchanged.

C CASE STUDY

  • Canary-0091 contains the coined medication Velfazvastatin, appears 64 times in training, and was selected as representative because its Qwen3.5-4B-Base perplexities were closest to median results across metrics.Table 5 presents transformed records, medication and name retention counts, and privacy leakage results.Source: C CASE STUDY · p. 22
  • The privacy-aware summary prompt combines the generic summary instruction with the privacy-aware rewriting instruction, and summary-based methods expand summaries using two format exemplars from fresh Synthea seeds.The exemplars are not drawn from the private cohort.Source: Table 4 · p. 24
  • The Table 5 examples show one processed copy; rewriting and summary methods process all 64 copies independently, with the displayed draw matching the majority handling of the coined medication.Goldfish alone excludes shaded tokens from the loss; PII scrubbing changes only the header.Source: Table 5 · p. 25
  • The displayed Qwen3.5-4B privacy-aware rewrite still includes a Social Security Number and a full street address.Source: C CASE STUDY · p. 27

Related papers

Loading 2610.22659v1…