PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Information Leakage of Updates to Natural Language Models

Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, Marc Brockschmidt

arXiv:1912.07942v4cs.LGcs.CLcs.CRstat.ML

TL;DR

Language-model updates and specialization can expose information about data added to or removed from training, creating privacy risks for models trained on private text. The paper compares model snapshots to analyze this leakage and reports that even tiny changes can reveal specific training text.

  • Problem

    The paper studies what an adversary with access to multiple model snapshots can learn about differences in their training data, a concern as high-capacity language models are fine-tuned on smaller private datasets.

    Source: §1 · p. 1

  • Method

    The paper analyzes differences between model snapshots and uses differential rank to approximate how highly sequences score relative to other candidate sequences.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Changes as small as 0.0001% of the original dataset can yield specific sentences or discourse fragments, and added or removed data can be extracted in the studied threat model.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Model updates pose a privacy threat that should be considered across the lifecycle of machine-learning applications.

    Source: §8 · p. 12

    Common words can cause score dips that make phrases harder to recover, requiring wider beams or searches looking more than one token ahead.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---\emph{differential score} and \emph{differential rank}---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.

1 INTRODUCTION

Comparing language-model snapshots can expose data added to or removed from training, even when the update is tiny. The paper studies this risk and finds that mitigation choices involve trade-offs.

1 INTRODUCTION

  • Differential scores compare sequence probabilities across snapshots, while beam search identifies sequences that may reveal training-data differences without background knowledge.The study tests canaries and real-world text across update training types, data proportions, adversary knowledge, dataset sizes, and RNN and transformer models.Source: §1 · pp. 1–2
  • The attack can recover a rare update phrase that updated-model-only search does not surface among its top 10,000 results.The phrase occurred six times in 25K update tokens and none in the original Reddit comments.Source: §1 · p. 2
  • Two-snapshot analysis can expose added or removed training data, including specific sentences, with changes as small as 0.0001% of the original dataset.Partial knowledge about the data can expose still smaller changes.Source: §1 · pp. 1–2
  • The study finds that analyzing two snapshots reveals more about added or removed data than analyzing a single snapshot at a time.The comparison is with single-snapshot analysis as in [6].Source: §1 · p. 2
  • Adding or removing non-sensitive training data is not a reliable mitigation, while differential privacy reduces attack effectiveness at substantial computational and model-utility cost.Restricting model access and truncating prediction outputs is promising because it reduces attack effectiveness without reducing utility; findings cover fine-tuning and retraining on combined data.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define autoregressive language models and the paper’s snapshot-attacker setting, then describe three update scenarios in which model comparisons may reveal training data.

2.1 Generative Language Models

  • Autoregressive models assign sequence probability through next-token probabilities; RNNs and transformers implement this differently but behave nearly identically for this analysis.The paper also introduces perplexity as a measure of how well a model matches test data, with lower values indicating a better match.Source: §2.1 · p. 2

2.2 Adversary Model and Goals

  • A model update changes parameters through training on different data, and each resulting version is a snapshot.Source: §2.2 · p. 2
  • A snapshot attacker queries two models trained on nested datasets and seeks to infer points in their difference, aiming in the best case to recover exact training examples.Snapshot attacks are reconstruction attacks; membership inference [25] [26] asks only whether a point was present, while model inversion targets attributes and aggregate statistics.Source: §2.2 · pp. 2–3
  • Differential privacy can protect groups of updated training points, but meaningful protection also limits how much more useful the updated model can be.The paper describes its practical effect as an empirical mitigation rather than a strong formal guarantee.Source: §2.2 · p. 3

2.3 Analysis Scenarios

  • The scenarios are data updates, specialization of a public pretrained model, and deletion; comparing snapshots can reveal update sentences, private specialization data, or deleted user data.For deletion, the older model may persist through slow downloads or copies, and the paper raises whether deletion could make data more susceptible to leakage.Source: §2.3 · p. 3

3 NEW METRICS

The section introduces differential score and differential rank to compare language-model snapshots, then approximates differential rank with beam search. The approximation can be exact with sufficiently large beams, while smaller beams provide a lower bound.

3 NEW METRICS

  • The paper introduces differential score and differential rank as metrics for analyzing data exposure between two generative language-model snapshots.Source: §3 · p. 3

3.1 Differential Score and Differential Rank

  • Differential score sums contextualized per-token probability differences, while relative differential score uses relative probability changes and is more robust to model-initialization noise.Source: §3.1 · pp. 3–4
  • Differential rank counts same-length sequences with higher differential scores; lower ranks indicate greater exposure, with rank 0 denoting the most exposed sequence.Source: §3.1 · p. 4

3.2 Approximating Differential Rank

  • Exact rank computation searches a space of size |T|^n, so beam search retains high-scoring candidate sequences and estimates rank from the returned set.Source: §3.2 · p. 4
  • A sufficiently large beam yields the true rank, whereas a smaller beam gives a lower bound; shrinking beam width during search can speed computation without compromising result quality in experiments.Source: §3.2 · p. 4
  • Group beam search divides initial one-token sequences into score-based groups and searches them independently to increase diversity among returned sequences.Source: §3.2 · p. 4

4 LEAKAGE ANALYSIS

Across canary and real-world updates, differential analysis of model snapshots exposes information about changed training data; experiments vary data, model, update method, and adversary knowledge.

4.1 Datasets and Models

  • The study uses PTB, Reddit, and Wikitext-103 with model configurations matched to low-, intermediate-, and large-data settings.The Reddit experiments use both an LSTM RNN and a Transformer; Wikitext-103 tests a large dataset with a relatively low-capacity model.Source: §4.1 · pp. 4–5

4.2 Implementing Model Updates

  • The experiments compare retraining from scratch with continued training and examine leakage under different data mixtures and levels of adversary background knowledge.The research questions include whether additional non-sensitive data masks private data and how training type affects leakage.Source: §4.2 · p. 5, §4.3 · p. 5

4.4 Results with Canaries

  • Across most canary types and insertion counts, the phrase ranks first in the beam-search results; rarer-token phrases are more exposed than high-to-low-frequency phrases.More insertions generally increase differential scores, while descending-frequency canaries tolerate more insertions before exposure.Source: §4.4 · pp. 5–6
  • At just 0.0001% of dataset tokens, inserted canaries yield differential scores approaching 4, near the 5-token canary upper bound.Table 1 summarizes scores across datasets, architectures, canary types, and insertion frequencies.Source: Table 1 · p. 6, §4.4 · p. 6
  • The proportion of added public data does not significantly affect canary leakage or the relative scores of the real-world phrases studied.The canary analysis also finds canaries extractable when contained in a substantially larger dataset extension.Source: §4.4 · p. 6, §4.5 · p. 7
  • Background prefixes expose later all-low-frequency canary tokens with little context, whereas high-to-low-frequency canaries require more context before scores rise.Differential rank may help complete a partially known phrase or confirm that it was used in an update.Source: §4.4 · p. 6

4.5 Results with Real-world Data

  • For real-world updates, group beam search returns on-topic phrases, and a sufficiently long known prefix can guide extraction of the complete phrase.The on-topic sequences are obtained without adversary knowledge of the dataset distribution or target information.Source: §4.5 · p. 7
  • For literal phrases, continued training scores fall as more fine-tuning data is added; for other phrases, its advantage over retraining can disappear or reverse.Continued training produces some phrases with scores too low to find after retraining, while the converse occurs more rarely and less consistently.Source: §4.5 · p. 7

5 CHARACTERIZING THE SOURCE OF LEAKAGE

The section examines whether leakage from model updates reflects overfitting or memorization, and whether access to two snapshots helps identify update data. Differential analysis surfaces sequences more associated with the update than single-model analysis, while the authors argue that memorization alone does not explain the findings.

RQ4: How important is access to a second model snapshot?

  • Table 5’s beam-search rank is heuristic: rank 0 indicates full-phrase recovery, but a phrase may not appear in the search results despite a favorable rank.For “Turkey searched an American plane,” all but seven no-prompt search results have higher rank (lower score).Source: Table 5 · p. 9
  • The analysis uses 3-gram-model perplexity on Reddit base data and talk.politics.mideast update data to compare extracted sequences’ dataset likelihoods.The RQ4 comparison asks whether differential analysis outputs are more likely to be in the update, uncommon there, and more so than single-model outputs.Source: §5 · pp. 8–9
  • Differentially extracted sequences were more likely to belong to the update than the base data, unlike sequences from single-model analyses; continued training also showed some of this effect in single-model results.For retraining from scratch, single-model outputs showed little significant difference; differential analysis produced sequences more likely to be from the update.Source: §5 · p. 9

RQ5: Is leakage due to overfitting or intended memorization?

  • Overfitting was curtailed with early stopping on a separate validation set, while the authors argue unintended memorization cannot alone explain canary recovery in a low-capacity, large-data setting.They identify amplification of otherwise imperceptible differences between snapshots as the most likely explanation.Source: §5 · p. 9
  • Extracted phrases generally matched the update dataset more closely, and complete sentences occurring as few as three times could be recovered.The authors report that intended memorization is unlikely to explain the results.Source: §5 · p. 9

6 MITIGATIONS

The paper evaluates differential privacy, two-stage continued training, and truncated outputs as mitigations for leakage from model updates. Truncation and additional training reduce leakage in the reported settings, while differential privacy mitigates the tested attack at substantial utility and computational cost.

6 MITIGATIONS

  • The paper considers differential privacy, continued training with public data, and truncating updated-model outputs as mitigation strategies.For protecting individual sentences, sequence-level privacy may suffice; protecting all update information may call for group DP.Source: §6 · p. 9, §6.1 · p. 10

6.1 Mitigation: Differential Privacy

  • The DP experiment used sequence-level privacy on PTB models with ε values of 5 and 111; next-token validation accuracy fell from about 23% to 11.89% and 13.34%, respectively.The tested models used a Gaussian noise mechanism with differentially private stochastic gradient descent.Source: §6.1 · p. 10
  • Sequence-level DP guarantees do not formally apply when canary phrases are inserted as multiple sequences, and the tested ε values are high.The authors note that the ε analysis is an upper bound.Source: §6.1 · p. 11
  • The differential rank of the canary’s two-token prefix rose from 0 to 9,458,399 (ε=5) and 849,685 (ε=111), but DP substantially degraded model performance and added substantial clipping overhead.The whole-phrase differential score fell from 3.94 to 4.5 × 10^-4 (ε=5) and 2.1 × 10^-3 (ε=111).Source: §6.1 · p. 11

6.2 Mitigation: Two-stage Continued Training

  • After a second continued-training stage, the canary’s differential score dropped substantially, suggesting multi-stage training with release of only the last model may mitigate leakage.The setup adds training on another dataset after training on the canaries.Source: §6.2 · p. 11

6.3 Mitigation: Truncating Output

  • Decreasing the updated model’s output limit moves extracted sequences closer to equal likelihood under the base and update data; truncation can reduce leakage without reducing output utility where realizable.For the same limit, sequences after continued training were more likely to be private than those after retraining from scratch.Source: §6.3 · p. 11

7 RELATED WORK

Related work studies leakage from single model snapshots, defenses, and other attack settings. The authors distinguish their snapshot-based recovery of update sequences from prior membership, memorization, and reconstruction approaches.

Membership inference attacks

  • Unlike membership inference that reports whether data is present, the approach recovers full sequences from differences between two snapshots without auxiliary data, as compared with Song and Shmatikov [27].The authors note that infrequent-token sequences provide a stronger signal to the adversary or auditor.Source: §7 · p. 11

Reconstruction attacks

  • The paper frames its attacks as reconstruction of update data using the original model as auxiliary information.It contrasts with Salem et al., whose attacker uses same-distribution auxiliary data and whose setting focuses on fine-tuning classification models.Source: §7 · pp. 11–12
  • Compared with Carlini et al. [6], the attack uses two model snapshots and data-distribution-following canaries, which can be extracted without context even at token frequency one in a million.Carlini et al. [6] studied canaries in a fixed context in financial-news data.Source: §7 · p. 11

Differential Privacy

  • Prior defenses include user-level DP for LSTMs [21] and DP protection of canaries [6]; the paper also distinguishes pan-privacy’s concern with observing internal states between updates.The cited work also reports an audit failure under user-level DP [27].Source: §7 · p. 12

Deletion of Data

  • Exact deletion is equivalent to retraining from scratch, so releasing snapshots before and after deletion falls within the paper’s attack setting.The paper leaves susceptibility of approximate-deletion models to snapshot attacks as an open question [15] [16].Source: §7 · p. 12

8 CONCLUSION

The conclusion treats model updates as a privacy threat that should be considered across the machine-learning lifecycle and calls for research to quantify and reduce unintended leakage.

8 CONCLUSION

  • The authors conclude that updates pose a lifecycle privacy threat and call for research to quantify and reduce unintended leakage from model updates.They describe the paper as a first systematic study of releasing snapshots of language models trained on overlapping data.Source: §8 · p. 12

A RESULTS FOR TALK.POLITICS.MIDEAST

Table 7 reports top-ranked length-4 phrases from a five-group beam search on talk.politics.mideast for RNN and Transformer models. The accompanying text notes that prolific poster Center for Policy Research discussed Turkey–Armenia tensions in posts around the dataset’s collection.

A RESULTS FOR TALK.POLITICS.MIDEAST

  • Table 7 lists highest-scoring length-4 sequences from a five-group beam search on talk.politics.mideast for RNN and Transformer architectures.The table is deferred from Section 4.5.Source: A RESULTS FOR TALK.POLITICS.MIDEAST · p. 13, Table 7 · p. 13
  • Around the 20 Newsgroups dataset’s collection, prolific poster Center for Policy Research discussed tensions between Turkey and Armenia.Source: Table 7 · p. 13

Related papers

Loading 1912.07942v4…