PaperlayerView exact revision on arXiv

Source-linked AI summary

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song

arXiv:1802.08232v3cs.LGcs.AIcs.CR

TL;DR

Rare or unique sequences in generative-model training data may be unintentionally memorized and exposed, creating privacy risks for models trained on sensitive text. The paper develops canary-based exposure testing and extraction procedures, finding memorization commonplace and difficult to prevent while demonstrating practical use in Smart Compose.

  • Problem

    Generative sequence models trained on sensitive text may disclose rare or unique training-data sequences, creating a need to measure unintended memorization and exposure risk.

  • Method

    The paper inserts random canary sequences at varying frequencies, measures their exposure through perplexity comparisons, and uses exposure-guided shortest-path search for extraction.

  • Results

    Unintended memorization is commonplace and hard to prevent, and the authors extract secret sequences such as credit card numbers from models trained without memorization safeguards.

  • Takeaways & Limitations

    Exposure testing helps practitioners choose training approaches that reduce privacy risk, including in the deployed commercial setting of Google Smart Compose.

  • Takeaways & Limitations

    The extraction algorithm can enumerate all non-leaf nodes in the worst case, although empirically it enumerates 3 to 5 orders of magnitude fewer nodes.

Abstract

from arXiv · show

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

1 Introduction

Neural sequence models can unintentionally memorize and disclose rare, sensitive training sequences. The paper introduces exposure-based testing to compare training strategies and support privacy-conscious model selection.

  • Motivation: Generative text models can disclose sensitive training sequences when users provide revealing prefixes and probe completion outputs.Such disclosures may expose secrets or uncommon word combinations, including valid-looking social-security numbers and business-related information.
  • Testing methodology: The paper introduces a quantitative exposure metric and canary-based testing methodology for measuring unintended memorization of rare sequences.Random canaries are inserted at varying frequencies, and exposure compares their perplexity with equivalent non-inserted sequences.
  • Results: An exposure-guided extraction algorithm can efficiently recover secret sequences that models consider unlikely, including credit card numbers from an Enron-trained language model.The authors present this as evidence that unintended memorization is a practical privacy concern rather than only an academic issue.
  • Results: Unintended memorization is commonplace and difficult to prevent: it occurs early, persists across models and training strategies, and survives rare data and relatively small models.Early stopping and dropout are insufficient; differential privacy eliminates the issue at some utility cost.
  • Results: Two training strategies can achieve near-identical accuracy while differing substantially in canary memorization, with Strategy A exposing a canary inserted 9 times.This illustrates why privacy decisions should consider empirical disclosure risk rather than training-data sensitivity alone.

2 Background: Neural Networks

Neural networks learn parameterized functions from examples through gradient-based optimization, while generative sequence models assign conditional probabilities to token sequences. Overtraining can impair generalization, but the paper distinguishes it from the memorization studied here.

  • Neural networks: A neural network is a parameterized function whose architecture defines computation structure and whose parameters θ are learned from training examples.Training data specifies what the network should compute when an explicit formulation is difficult.
  • Training: Gradient descent updates network parameters using minibatches, a loss function, and learning rate η to reduce prediction error.The training update uses the current weights and examples sampled from the training set.
  • Training: Training across multiple epochs can expose models repeatedly to potentially sensitive examples.An epoch is one iteration over the entire training set.
  • Sequence models: Generative sequence models estimate Pr(x1...xn) by decomposing it into conditional token probabilities Pr(xi|x1...xi−1).Neural networks commonly produce these conditional distributions from preceding tokens.
  • Overtraining: At epoch 16 in the illustrated overtraining example, validation loss begins increasing after initially decreasing, indicating reduced generalization and increased label memorization.Training loss continues decreasing while validation loss rises.
  • Overtraining: The paper reports that its memorization results are not caused by overtraining: uncommon random data is memorized throughout learning, well before maximum utility.It therefore uses “overtraining” rather than “overfitting” for the point where validation loss stops decreasing.

3 Do Neural Nets Unintentionally Memorize?

The paper defines unintended memorization as revealing out-of-distribution training data unrelated to the learned task. A concrete language-model experiment shows that a rare inserted social-security number can be recovered from a partial prefix.

  • Definition: Unintended memorization occurs when a trained neural network may reveal out-of-distribution training data that is irrelevant and unhelpful to the learning task.The paper operationalizes such data as secrets and tests it by inserting random sequences as canaries.
  • Motivating example: The motivating risk is that sentence-completion models may emit rare sensitive information from one user when another user supplies the corresponding prefix.The paper frames non-emission of exact sensitive suffixes as the desired behavior.
  • Experiment: A character-level LSTM trained on the Penn Treebank dataset was augmented with one out-of-distribution sentence containing a social-security number.The model had two LSTM layers, 200 hidden units, and approximately 600,000 parameters, and was trained without overtraining.
  • Results: A greedy or beam search recovered the complete inserted social-security number after receiving the prefix “My social security number is 078-”.The result held across repeated experiments, with prefixes containing the first two to four digits yielding the remaining digits.
  • Implication: These results motivate the exposure metric and its associated testing methodology for measuring disclosure risk.The paper presents the example as evidence that neural-network training can produce exact secret completions unless care is taken.

4 Measuring Unintended Memorization

The paper measures unintended memorization by comparing how a model ranks inserted canaries against alternative sequences, then defines exposure as the information gained for guessing a canary. Because exact rank is expensive, it develops efficient approximations and validates one against the measured distribution.

  • Notation and Setup: Log-perplexity measures sequence likelihood, but its absolute value depends heavily on the model, application, and dataset.Lower perplexity indicates that the model is less surprised by the sequence.
  • Notation and Setup: Canaries are randomly instantiated sequences formed by filling format holes from a randomness space R.For example, a digit format can be filled with a specific random number.
  • The Precise Exposure Metric: Rank orders all possible instantiated canaries by model perplexity, placing more likely sequences earlier.In the example, the inserted canary 281265017 has rank 1 because no candidate has lower perplexity.
  • The Precise Exposure Metric: Rank is computationally expensive because it requires evaluating every possible candidate, motivating an efficiently approximated exposure measure.The approximation targets how model access reduces the work needed to guess a secret.
  • The Precise Exposure Metric: Exposure quantifies the reduction in guessing entropy, equivalently reflecting how much a model improves guesses about a randomly chosen canary.Its definition adds a constant based on |R| to negative log-rank, keeping exposure nonnegative.
  • The Precise Exposure Metric: Exposure ranges from 0 to log2 |R|, with the maximum for the top-ranked canary and the minimum for the least likely canary.The metric is not normalized, so its magnitude depends on the search-space size.
  • Efficiently Approximating Exposure: The sampling approximation can distinguish a marginally most-likely canary from one that is far more likely, despite the estimate having no upper bound.The true exposure remains upper-bounded by log2 |R|.
  • Efficiently Approximating Exposure: A skew-normal approximation nearly matches the discrete distribution, with a Kolmogorov–Smirnov test failing to reject equality at p > 0.1.The comparison uses 106 samples from the measured canary distribution.

5 Exposure-Based Testing Methodology

The testing methodology inserts random out-of-distribution canaries into training data, trains a model under the intended setup, and measures exposure to assess unintended memorization. It supports comparisons across insertion frequencies, randomness choices, and training approaches.

  • Methodology: The methodology uses exposure to test memorization in research datasets and production systems, including Google’s Smart Compose.The authors describe the approach as simple and effective.
  • Methodology: Measured exposure helps practitioners make informed decisions about whether stronger privacy defenses are needed under different training settings.The methodology bases decisions on empirically observed memorization.
  • Methodology: Out-of-distribution canaries isolate unintended memorization because they are unrelated to and unhelpful for the learning task.In-distribution phrases might instead be desirably memorized because they support the task.
  • Methodology: The canary format usually has little effect, but the randomness space should match the testing objective.Highly out-of-distribution canaries approximate worst-case bounds, while in-distribution canaries target more average-case bounds.
  • Methodology: Researchers insert randomly chosen canaries at varied frequencies, including once, tens of times, or hundreds or thousands of times.Varying insertion counts enables exposure to be plotted against the number of insertions.
  • Methodology: The test model should use the same architecture, optimizer, iteration count, and hyperparameters intended for final training.These choices can affect the amount of memorization, so testing must match the practical setup.

6 Experimental Evaluation

Across language-modeling and translation experiments, the exposure methodology reveals substantial variation in unintended memorization across architectures, datasets, and training choices. The experiments show that memorization can occur at low insertion frequencies, while exposure-based testing helps identify privacy-preserving operating points.

  • Experimental scope: The methodology is applied across architectures and datasets to evaluate exposure and demonstrate that unintended memorization is common.Experiments include Smart Compose, word-level and character-level language models, and neural machine translation.
  • Smart Compose: Smart Compose testing measures exposure in a model trained on potentially sensitive email data and supports limiting learned information to phrases used by multiple users.The model is a large LSTM trained on billions of word sequences, and exposure results are used to avoid exposing private sequences from individual users.
  • Smart Compose: When secrets occur once in a billion examples, exposure is negligible; at higher frequencies, inserted canaries become 1,000× more likely than non-inserted canaries but remain difficult to extract.The higher exposure still does not enable extraction by the paper’s algorithms, much less accidental discovery.
  • Word-level language model: Word-level models trained with different hyperparameters show large memorization differences, and models with similar utility can exhibit less memorization.The paper presents utility–exposure trade-offs using a Pareto frontier; memorization and utility are not highly correlated (r=-0.32).
  • Character-level language model: Character-level models memorize inserted random words less readily than word-level models, reaching exposure 60 after 16 insertions, which remains insufficient for extraction.The comparison concerns random words; the paper reports stronger memorization for random number sequences in the character-level setting.
  • Neural machine translation: In neural machine translation, inserting a canary once makes it 1,000× more likely than random chance, and inserting it four times makes it completely memorized.The translation setting uses a paired English–Vietnamese canary and a task-specific perplexity measure.

7 Characterizing Unintended Memorization

Unintended memorization begins early in training, fluctuates with canary-containing batches, and is not simply a consequence of prolonged overtraining. Exposure increases during learning and then stabilizes or declines even as training continues.

  • 7.1 Memorization Throughout Training: Exposure spikes whenever a mini-batch contains the canary and fluctuates or sometimes decreases during other mini-batches.This pattern appears during the first three epochs of training on 10% of the training data.
  • 7.1 Memorization Throughout Training: After one epoch, exposure is 3, making the canary 8× more likely than an equivalent random sequence; after three epochs, exposure is 8.At exposure 8, access to the model reduces the number of guesses needed to guess the canary by over 100×.
  • 7.2 Memorization and Overtraining: Exposure peaks around epoch 10, when testing loss is minimized, and does not continue increasing as training proceeds into overtraining.The model’s training loss continues to decrease after testing loss begins increasing.
  • 7.2 Memorization and Overtraining: Estimated exposure at epoch 10 is higher than at epoch 40 (p < .001), although the canary’s rank remains 1 after epoch 10.Thus, continued training changes the exposure estimate without changing the canary’s top rank.
  • 7.2 Memorization and Overtraining: Together, the results indicate that exposure increases while the model is learning and not while learning has stopped, suggesting memorization is a component of training.The paper connects this observation to prior claims that neural networks first minimize training loss by memorizing training data.

8 Validating Exposure with Extraction

The paper validates exposure by testing whether high-exposure canaries can be efficiently extracted. A Dijkstra-inspired shortest-path search makes extraction practical, and experiments show extraction success tracks exposure, including for naturally occurring secrets.

  • 8 Validating Exposure with Extraction: Exposure is validated by attempting to extract inserted canaries when exposure exceeds log2 |R |.The extraction algorithm tests whether the metric correctly identifies memorized sequences.
  • 8 Validating Exposure with Extraction: Brute-force extraction enumerates all candidate sequences by perplexity, but searching credit-card numbers would require 4,100 commodity GPU-years.The baseline becomes impractical when the candidate space is large.
  • 8 Validating Exposure with Extraction: A Dijkstra-inspired shortest-path algorithm reduces extraction complexity by several orders of magnitude by searching a weighted tree of partial strings.Edge weights are negative log-likelihoods, so the lightest root-to-leaf path corresponds to the lowest-perplexity sequence.
  • 8 Validating Exposure with Extraction: 105 total queries recover a fully memorized canary, four orders of magnitude fewer than brute force.The canary had exposure above 30 and was verified to have the lowest perplexity among 10^9 candidates.
  • 8 Validating Exposure with Extraction: Extraction is always possible when exposure exceeds 33 and never possible when exposure is below 31.The experiment varied canary insertion counts and training regimes while holding final test accuracy constant.
  • 8 Validating Exposure with Extraction: Three naturally occurring Enron secrets were extractable, with credit-card and social-security numbers recovered in a few hours of local computation.The reported local runtime batches model queries and excludes remote-querying time, so it is not an estimate of actual attack duration.

9 Preventing Unintended Memorization

The paper evaluates regularization, sanitization, and differential privacy as defenses against unintended memorization. Regularization and blacklist-style sanitization do not reliably prevent extraction, whereas DP-SGD eliminates memorization in the reported experiment with a limited utility cost.

  • 9 Preventing Unintended Memorization: Weight decay, dropout, and quantization do not prevent inserted canaries from being extracted.These regularization approaches are intended to reduce overtraining, but the experiments find no corresponding protection against memorization.
  • 9 Preventing Unintended Memorization: Weight decay lowers validation loss in an overtraining setting but has no effect on canary exposure.On the original model, weight decay produced neither validation-loss improvement nor memorization reduction.
  • 9 Preventing Unintended Memorization: Dropout does not statistically significantly reduce unintended memorization, while rates above 30% reduce test accuracy in the reported model.The tested dropout range was 0% to 90%, with ten models trained at each rate.
  • 9 Preventing Unintended Memorization: Blacklist-style sanitization cannot guarantee removal of all sensitive sequences, and the evaluated two-model method missed some secrets.The approach is also unsound when the same secret is inserted twice.
  • 9 Preventing Unintended Memorization: DP-SGD fully eliminates the memorization effect in the experiment, with the highest-utility private model showing 10% higher test loss than the non-private baseline.The method clips per-example gradients and adds Gaussian noise; at ε = 1.0, exposure drops to 1.
  • 9 Preventing Unintended Memorization: The reported privacy analysis has a growing gap between measured privacy loss and the upper-bound ε DP guarantees at very large ε.The authors note that tighter analyses can improve guarantees but may require additional assumptions.

10 Related Work and Conclusions

The paper distinguishes unintended memorization from related privacy attacks and introduces exposure to quantify memorization and test potential training-data disclosure. It finds memorization can persist despite conventional defenses, while the approach remains limited in scope and assumptions.

  • Related Work: Exposure quantifies how much memorization occurred, rather than only testing whether a training example was a member.
  • Limitations and Future Work: The study is limited to generative models, while other model types require further work, and its extraction algorithm assumes access to likely outputs, context, and possible canary values.
  • Limitations and Future Work: The exposure computation uses input-output behavior, although white-box access to weights and activations might enable stronger memorization measures.
  • Conclusions: The paper studies unintended memorization of rare training details in generative models while they are still learning the intended behavior.
  • Conclusions: Unintended memorization remains a concern even without overtraining, including when rare examples appear only a handful of times and are outliers.
  • Conclusions: The testing methodology inserts canaries into training data and uses exposure to directly measure their memorization.
  • Conclusions: The technique supports decisions throughout training, including data curation, architecture and hyperparameter selection, and interpreting differentially private training results.

A.1 Across Different Architectures

Across architectures with equal theoretical capacity, all evaluated models unintentionally memorize canaries. LSTM and GRU achieve the highest accuracy and exposure, while CNNs perform worse and memorize less.

  • LSTM and GRU show both the highest accuracy (lowest loss) and the highest exposure.
  • CNNs have lower accuracy and lower exposure than the recurrent architectures.

A.2 Across Training Strategies

Training choices affect exposure even when models achieve similar task performance. Larger batches and models increase memorization, while smaller batches reduce it at a substantial training-speed cost; optimizer choices show no significant exposure difference after accuracy control, and added canary context improves detection.

  • Batch Size: Larger batch sizes exhibit significantly more memorization even when models with the same capacity reach nearly identical training and testing loss.The batch-size evaluation ranges from 16 to 1024 and averages ten models for each setting.
  • Batch Size: Reducing batch size can reduce memorization but may be prohibitively slow because it limits parallelized training across GPUs and servers.
  • Shuffling, Bagging, and Optimization Method: After controlling for final test accuracy, optimizer, shuffling, and bagging settings show no statistically significant difference in canary exposure.
  • Batch Size: Larger batch sizes and larger models both increase memorization.
  • Context and Canary Format: Exposure is computed at epochs 5 and 10 before the models completely memorize the inserted canary.
  • Context and Canary Format: Additional knowledge about a canary’s format and surrounding context increases the ability to detect memorization, especially when inner context is available.The effect of additional context becomes increasingly important as training proceeds.
Loading 1802.08232v3…