PaperlayerView exact revision on arXiv

Source-linked AI summary

Defending against Reconstruction Attacks with R\'enyi Differential Privacy

Pierre Stock, Igor Shilov, Ilya Mironov, Alexandre Sablayrolles

arXiv:2202.07623v1cs.LGcs.AIcs.CRstat.ML

TL;DR

Reconstruction attacks can recover training samples from trained language models, while conventional privacy budgets may not provide meaningful protection. The paper derives reconstruction guarantees from RDP and finds that membership inference can remain possible even as extraction of sufficiently high-entropy secrets stays difficult.

  • Problem

    Language models can emit verbatim training samples, and commonly used differential-privacy budgets may not yield meaningful guarantees against reconstruction.

    Source: §1 · p. 1

  • Method

    The paper uses RDP probability-preservation guarantees to analyze reconstruction of secrets, including attacks composed with the training mechanism.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    RDP gives better reconstruction guarantees for the same mechanism, with an intermediate regime where membership inference is not protected but full reconstruction of sufficiently high-entropy secrets remains difficult.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Secret length itself provides protection against reconstruction, so membership inference and extraction can have different privacy outcomes.

    Source: §1 · p. 2, §3.1 · p. 5

    For practical reconstruction attacks, secret randomness is difficult to estimate, and shared sequence structure can make absent sequences more likely after related patterns appear in training.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Reconstruction attacks allow an adversary to regenerate data samples of the training set using access to only a trained model. It has been recently shown that simple heuristics can reconstruct data samples from language models, making this threat scenario an important aspect of model release. Differential privacy is a known solution to such attacks, but is often used with a relatively large privacy budget (epsilon > 8) which does not translate to meaningful guarantees. In this paper we show that, for a same mechanism, we can derive privacy guarantees for reconstruction attacks that are better than the traditional ones from the literature. In particular, we show that larger privacy budgets do not protect against membership inference, but can still protect extraction of rare secrets. We show experimentally that our guarantees hold against various language models, including GPT-2 finetuned on Wikitext-103.

1. Introduction

Language models can reproduce verbatim training sentences, motivating privacy protections for model releases. The paper uses RDP to derive reconstruction guarantees and argues that secret length can protect extraction even when membership inference remains possible.

1. Introduction

  • RDP can leave membership inference unprotected while making full reconstruction of sufficiently high-entropy secrets difficult.The distinction is that an attacker may test a known secret’s membership without being able to extract it when they do not already know it.Source: §1 · p. 1
  • Secret secrecy is measured in unknown bits b, with an average of 2^b trials needed to guess a sample generated with probability 2^-b.The number of trials corresponds to the verification cost an adversary may pay, such as dialing a guessed phone number.Source: §1 · p. 1
  • Across the plotted secret lengths and DP-noise levels, the theoretical bound prevents total disclosure, and is comparatively better at lower noise.The figure compares the leakage bound with the y = x total-leakage threshold.Source: Figure 1 · p. 2
  • The paper derives stronger secret-protection guarantees from RDP probability preservation, with secret length itself providing additional protection.RDP guarantees hold despite prior knowledge, according to the introduction.Source: §1 · p. 2
  • For n-gram language models, experiments show the proposed guarantee is tight; in the reported fine-tuning attack, observed leakage is smaller than the guarantee.The fine-tuning claim is limited to the stated attack model.Source: §1 · p. 2

2. Background

The background distinguishes attacks that test membership from extraction attacks that recover training text. It reviews DP and RDP, then describes DP-SGD and prior private language-model fine-tuning work.

2.1. Privacy Attacks

  • Membership inference tests whether a given sample was in training, while language-model extraction seeks verbatim training sentences from generative models.Extraction work described here uses canaries to measure exposure and has also recovered training sentences from GPT-2.Source: §2.1 · p. 2
  • Recent membership attacks focus on confidently identifying a few members, using true- and false-positive rates and calibrated loss scores.The cited approaches center loss or fit a Gaussian likelihood to improve attack performance.Source: §2.1 · p. 2
  • The study focuses on uncommon samples that a model overconfidently predicts.The text connects these samples to the focus of the study.Source: §2.1 · p. 2

2.2. Differential Privacy

  • RDP is introduced for tighter composition analysis of the subsampled Gaussian mechanism, and its guarantees persist under post-processing.The text states that RDP guarantees can be translated into DP guarantees, but not conversely.Source: §2.2 · p. 3
  • Probability preservation means that an event’s probabilities under adjacent datasets are constrained by RDP; the guarantee also applies after a reconstruction attack is composed with training.The paper denotes the event probabilities as p and p′ and treats an attack on a trained model as post-processing.Source: §2.2 · p. 3
  • DP-SGD samples training examples, clips per-sample gradients using a constant C, adds noise, and tracks a privacy budget dependent on δ, sampling rate, steps, and noise multiplier.The passage describes sampling without replacement at rate q and using the resulting batch gradient in optimization.Source: §2.2 · p. 3

3. Our method

The method derives an RDP leakage bound for reconstruction attacks and uses lazy sampling to identify training samples likely to leak. It compares the bound with traditional guarantees and relates secret length and attack scope to reconstruction risk.

3.1. Reconstruction and Canaries

  • RDP bounds the change in a secret’s reconstruction probability between training with and without that secret, and post-processing preserves the guarantee for any reconstruction attack.The setup defines p0 as the prior probability and p1 as the posterior probability of reconstructing the secret.Source: §3.1 · p. 4
  • The authors report numerically better leakage bounds than Balle et al. (2020) and describe absolute leakage as nominally stronger because it also prevents the posterior from falling below the prior.Source: §3.1 · p. 4
  • At b = 40 bits and σ = 0.4, L(p0) stays partial as iterations grow while minα h(α, p0) becomes vacuous; the gap shrinks with larger σ.The paper also reports numerically better bounds than Balle et al. (2020).Source: Figure 2 · p. 4, §3.1 · p. 4
  • Longer secrets leak more in absolute terms but less in relative terms, and the upper bound on posterior probability is minimized at zero secret bits.The leakage function is non-decreasing and concave; the authors conclude that secret length itself protects against reconstruction.Source: §3.1 · pp. 4–5
  • Membership inference targets secrets with few bits, so some privacy settings may not prevent it while still limiting full reconstruction of rare, longer secrets.The cited discussion notes typical membership-inference priors near 1/2 and alternative ratios within a factor of 1 to 10.Source: §3.1 · p. 5

3.2. Lazy Sampling: Identifying Samples Likely to Leak

  • Lazy sampling estimates the samples needed to generate a secret as about 1/p when its probability under the attack is tractable.Language-model sequence probabilities factor into conditional token probabilities, including for temperature and top-k sampling; top-k can assign zero probability to a sequence.Source: §3.2 · p. 5
  • Lazy sampling does not analyze Internet-prompt sampling, and failure to reconstruct a sample with one attack does not imply that all attacks will fail.Internet sampling uses crawled public-web text as a prompt for iterative generation.Source: §3.2 · p. 5
  • The analysis assumes a secret’s probability varies little across models trained on the same dataset, then compares training-sample generation probabilities with filtering scores.In an n-gram experiment with 10,000 models, at σ = 2.875 the log-probability had mean ≃−22.5 and standard deviation ≃1.5.Source: §3.2 · p. 5

4. Experiments

Experiments compare RDP leakage bounds with empirical leakage in n-gram models, examine reconstruction risks without defenses, and test private GPT-2 fine-tuning and its limits.

4.1. n-gram Language Model

  • The RDP bound L(p0) was tight against empirical log(p1/p0) leakage for the n-gram language model.Figure 4 compares the empirical leakage with the theoretical bound.Source: Figure 4 · p. 6, §4.1 · p. 6

4.2. Lazy Reconstruction without Defenses

  • Training samples had lower calibrated loss than test samples, while target-generated sentences had lower average calibrated loss than training sentences.The comparison used three OpenWebText subsets of one million samples each for training, calibration, and testing.Source: §4.2 · p. 6, Figure 5 · p. 7
  • Training sentences with high target-model generation probability and low calibrated loss were most at risk; very-low-probability memorized sentences were unlikely to surface with this strategy.The analysis uses attack-model generation probability rather than perplexity and examines training-set sentences, unlike the analogous Carlini et al. (2021b) analysis.Source: §4.2 · p. 7

4.3. Real-life Canary Experiments

  • Private GPT-2 fine-tuning on Wikitext-103 reached perplexity 45, compared with 38 for non-private fine-tuning.The setup used a batch size of 1024, clipping threshold 1, and learning rate 0.0001.Source: §4.3 · p. 7
  • The GPT-2 canary was a prompt followed by 16 random digits, and the authors upsampled it up to 280 times relative to regular samples.They estimated the secret probability from the model's probability of the digits given the prompt.Source: §4.3 · p. 7
  • Without DP, the model predicted the canary with p1 = 67%; with DP, lower σ increased leakage, but σ = 0.3 did not fully reconstruct it.The canary was upsampled up to 280 times relative to regular samples to make the method more sensitive.Source: §4.3 · p. 7

4.4. Limits of Reconstruction Attacks

  • Practical reconstruction complicates leakage attribution because secrets may be duplicated or share structure that makes unseen combinations more likely.The paper notes that trigram-overlap deduplication does not account for knowledge shared across secrets.Source: §4.4 · p. 8

5. Conclusion

RDP with DP-SGD provides meaningful guarantees against reconstruction attacks. The conclusion links these guarantees to private language-model fine-tuning and distinguishes extraction protection from membership-inference protection.

5. Conclusion

  • RDP with DP-SGD provides meaningful guarantees against reconstruction attacks.The conclusion says these guarantees can be achieved with lower noise, or equivalently higher signal-to-noise ratios, which should translate to better accuracies.Source: §5 · p. 8
  • Combining the improved guarantees with private fine-tuning allows language models to reach perplexity competitive with the state of the art while retaining meaningful protection against training-data extraction.The conclusion names Li et al. (2021) as the private fine-tuning work used in this combination.Source: §5 · p. 8
  • Reconstruction attacks can be defeated at noise levels that would otherwise be vacuous for membership-inference defense.The conclusion notes that reconstruction attacks require access to a trained model, not candidate samples.Source: §5 · p. 8

A.1. Probability preservation

The passage presents a derivation from Eq. 1 to Eq. 2 as a two-sided bound on log(p1/p0).

A.1. Probability preservation

  • The passage gives steps for deriving Eq. 2 from Eq. 1.The resulting relation bounds log(p1/p0) from both below and above.Source: §A.1 · p. 11
  • The lower bound is −d_α + 1/(α − 1) log(p0) ≤ log(p1/p0).Source: §A.1 · p. 11
  • The upper bound is log(p1/p0) ≤ (α − 1)d_α/α − 1/α log(p0).Source: §A.1 · p. 11

A.2. Leakage function

The appendix establishes that the leakage function L2 is non-decreasing and concave, then uses these properties to show that ξ(b)=L2(b)−b is non-increasing.

A.2. Leakage function

  • The pointwise minimum of non-decreasing functions is non-decreasing, and the pointwise minimum of concave functions is concave.The appendix justifies the latter property by noting that the hypograph of the minimum is the intersection of the functions’ convex hypographs.Source: §A.2 · p. 11
  • The leakage function L2 is non-decreasing and concave because it is the pointwise minimum of linear, non-decreasing functions f_α.Source: §A.2 · p. 11
  • ξ(b)=L2(b)−b is non-increasing, with the appendix deriving this from its derivative being non-positive.Source: §A.2 · p. 11

Related papers

Loading 2202.07623v1…