PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Pretraining Data from Large Language Models

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, Luke Zettlemoyer

arXiv:2310.16789v3cs.CLcs.CRcs.LG

TL;DR

The paper asks whether text appeared in an LLM’s undisclosed pretraining data, a question complicated by unavailable training distributions and changing detection difficulty. It introduces WIKIMIA and the reference-free Min-K% Prob method, which averages probabilities for low-probability tokens. Min-K% Prob consistently outperforms baselines, including a 7.4% improvement over the best baseline on WIKIMIA, and supports audits of books, contamination, and unlearning.

  • Problem

    The paper addresses whether a given text belongs to an LLM’s pretraining data when only black-box model access is available and the training-data distribution is unavailable.

  • Method

    The paper introduces WIKIMIA, a temporally constructed benchmark, and Min-K% Prob, which averages the probabilities of a text’s lowest-probability tokens without additional training.

  • Results

    7.4% improvement over the best baseline method was achieved in AUC on WIKIMIA, while Min-K% Prob consistently outperformed baselines across diverse models and settings.

  • Takeaways & Limitations

    Min-K% Prob provides a reference-free approach applied to copyrighted-book detection, downstream-data contamination detection, and privacy auditing of machine unlearning.

  • Takeaways & Limitations

    Pretraining-data detection is constrained by unavailable training distributions and difficulty changes associated with dataset size, occurrence frequency, and learning rate.

Abstract

from arXiv · show

Although large language models (LLMs) are widely deployed, the data used to train them is rarely disclosed. Given the incredible scale of this data, up to trillions of tokens, it is all but certain that it includes potentially problematic text such as copyrighted materials, personally identifiable information, and test data for widely reported reference benchmarks. However, we currently have no way to know which data of these types is included or in what proportions. In this paper, we study the pretraining data detection problem: given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model was trained on the provided text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that uses data created before and after model training to support gold truth detection. We also introduce a new detection method Min-K% Prob based on a simple hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data. Moreover, our experiments demonstrate that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous methods. We apply Min-K% Prob to three real-world scenarios, copyrighted book detection, contaminated downstream example detection and privacy auditing of machine unlearning, and find it a consistently effective solution.

1 INTRODUCTION

The paper frames pretraining-data detection as a black-box membership question and introduces WIKIMIA and Min-K% Prob to address it without access to the training distribution. The method uses low-probability outlier tokens, improves over baselines on WIKIMIA, and is applied to several real-world auditing scenarios.

  • Motivation: Pretraining-data detection asks whether an LLM was trained on a given text using only black-box access and no knowledge of its pretraining data.The problem is motivated by undisclosed training sources and possible inclusion of copyrighted, private, or benchmark-related text.
  • WIKIMIA: WIKIMIA uses timestamped Wikipedia events to provide member data from before training and non-member data from after training.Its automated construction supports accurate, general, and continually updated evaluation across models that use Wikipedia.
  • Min-K% Prob: Min-K% Prob detects membership by averaging the log likelihoods of the k% lowest-probability tokens in a text.Its hypothesis is that unseen examples contain more low-probability outlier words than seen examples.
  • Results: 7.4% improvement over the strongest baseline in AUC was achieved by Min-K% Prob on WIKIMIA.The method requires neither pretraining-corpus knowledge nor additional training, unlike reference-model approaches.
  • Applications: The authors apply Min-K% Prob to copyrighted book detection, downstream-data contamination detection, and privacy auditing of machine unlearning.The case studies report evidence of copyrighted-book training, residual copyrighted outputs after unlearning, and training-factor effects on detection difficulty.

2 PRETRAININING DATA DETECTION PROBLEM

The paper defines pretraining-data detection as black-box membership inference under unavailable training-distribution information and distinctive pretraining-scale challenges. WIKIMIA addresses evaluation by using temporally separated Wikipedia data, while the benchmark also tests paraphrases and different text lengths.

  • Problem definition: Pretraining-data detection learns a detector that infers whether an arbitrary data point belongs to a language model’s training data.The detector has black-box access to the model and can compute token probabilities for candidate data points.
  • Challenge 1: Unavailability of the pretraining data distribution: Reference-model MIA methods require the target training-data distribution and enough shadow samples to train a calibration model.Those requirements are problematic when pretraining data are undisclosed and reference-model training is computationally expensive.
  • Challenge 2: Detection difficulty: Detection becomes harder as dataset size increases and training epochs or learning rates decrease.The paper relates this difficulty to similarity between member and non-member model outputs, whose total variation bound depends on occurrence frequency, learning rates, and inverse dataset size.
  • WIKIMIA benchmark: WIKIMIA treats Wikipedia events added after selected dates as non-members because they are guaranteed to be absent from earlier pretraining data.Member articles are collected from before 2017, while post-2023 events form the recent non-member set.
  • Evaluation settings: The benchmark evaluates both verbatim and paraphrased examples and examines detection across different text-length buckets.These settings address whether semantic equivalents can be detected and how length affects method difficulty and rankings.

3 MIN-K% PROB: A SIMPLE REFERENCE-FREE PRETRAINING DATA DETECTION METHOD

MIN-K% PROB detects pretraining membership without a reference model by focusing on the least probable tokens in a text. It thresholds the resulting score to classify examples as seen or unseen.

  • MIN-K% PROB uses minimum token probabilities to detect whether text was included in pretraining data.Non-member examples are hypothesized to contain a few low-probability outlier words, whereas member examples are less likely to do so.
  • For each token, the method computes its conditional log-likelihood given the preceding tokens.
  • The method selects the k% of tokens with the minimum probabilities and averages their log-likelihoods.
  • A text’s pretraining membership is determined by thresholding the MIN-K% PROB score.The selected set size is denoted E.

4 EXPERIMENTS

The experiments evaluate MIN-K% PROB and baseline membership-inference methods across language models, text lengths, and original or paraphrased examples. MIN-K% PROB consistently outperforms the baselines, averaging 0.72 AUC and improving 7.4% over PPL.

  • Baseline detection methods: The evaluation compares reference-based and reference-free membership-inference methods, including PPL, Neighbor, Zlib, Lowercase, and Smaller Ref.
  • Implementation details: k = 20 works best in a held-out sweep over 10, 20, 30, 40, and 50, and this value is used without further tuning.The sweep uses the LLAMA-60B model.
  • Main results: MIN-K% PROB consistently outperforms baseline methods across diverse language models in both original and paraphrase settings.The evaluation includes LLaMA, GPT-Neo, and Pythia models, with AUC and TPR@5%FPR as metrics.
  • Main results: 0.72 average AUC is achieved by MIN-K% PROB, a 7.4% improvement over the best baseline, PPL.PPL is the loss-attack baseline based on the target model’s example perplexity.
  • Model size: AUC increases with model size, likely because larger models have more parameters and are more likely to memorize pretraining data.
  • Length of text: AUC increases with text length, likely because longer texts contain more memorized information and are more distinguishable from unseen texts.

5 CASE STUDY: DETECTING COPYRIGHTED BOOKS IN PRETRAINING DATA

The copyrighted-book case study applies MIN-K% PROB to Books3 excerpts to identify possible GPT-3 pretraining contamination. It achieves 0.88 AUC, and nearly 90% of sampled books have detected contamination rates above 50%.

  • MIN-K% PROB is applied to Books3 excerpts to detect copyrighted material that may have been included in GPT-3’s training data.
  • Test data and metrics: The test set contains 10,000 randomly extracted 512-word snippets from 100 copyrighted Books3 books.For each book, the reported contamination rate is the percentage of snippets identified as pretraining data.
  • Results: 0.88 AUC is achieved by MIN-K% PROB, outperforming baselines in detecting copyrighted books.
  • Results: Table 2 lists the top 20 copyrighted books by predicted contamination rate in GPT-3’s pretraining data.

6 CASE STUDY: DETECTING DOWNSTREAM DATASET CONTAMINATION

This case study evaluates MIN-K% PROB for detecting downstream benchmark contamination and examines how training conditions affect detection difficulty. The method outperforms baselines, while dataset size has opposite effects for outlier contaminants and general in-distribution samples.

  • Experiments: MIN-K% PROB outperforms all baselines for detecting leaked downstream examples.The evaluation uses contaminated pretraining data and reports AUC over 400 examples.
  • Results and analysis: Detection becomes harder as contaminant occurrence frequency and learning rate decrease.Higher learning rates increase AUC scores across downstream tasks, consistent with stronger memorization.
  • Results and analysis: Outlier contaminant detection becomes easier with larger pretraining datasets, whereas general in-distribution detection becomes harder.The contrasting trends are attributed to memorization of long-tail samples for outlier contaminants.
  • Results and analysis: AUC scores positively correlate with the occurrence frequency of downstream examples.The study inserts multiple copies according to a Poisson distribution to measure this relationship.

7 CASE STUDY: PRIVACY AUDITING OF MACHINE UNLEARNING

The paper applies MIN-K% PROB to audit whether machine unlearning removed Harry Potter content from an LLM. Story completion and question answering both reveal residual similarities or knowledge in the unlearned model.

  • Method: MIN-K% PROB identifies suspicious Harry Potter text chunks for auditing the unlearned model.The study compares scores from the unlearned and original LLaMA2-7B-chat models over approximately 1,000 512-word chunks.
  • Story completion: 5.3% of generated story completions receive a GPT score of at least 4 for similarity to the gold continuation.The evaluation uses SimCSE and GPT-4 similarity scores.
  • Story completion: 10 chunks have GPT-4 similarity scores of at least 4, with all showcased examples exceeding 0.7 SimCSE similarity.The study uses only Harry Potter books 1–4, and the authors note that the full series could expose more failed chunks.
  • Question answering: The unlearned model correctly answers selected Harry Potter questions, including cases with ROUGE-L recall of 1.Responses were manually cross-checked against the book series.
  • Results: The findings suggest that Harry Potter knowledge was not completely erased from the unlearned model.This conclusion follows from both highly similar story completions and verified question-answering results.

8 RELATED WORK

Related work frames pretraining-data detection as a membership inference problem and reviews contamination-detection methods. Existing approaches commonly require training-corpus access or rely on model prompting and memorized outputs.

  • Membership inference attacks: Membership inference attacks determine whether a sample belonged to a model’s training data and can expose privacy risks.They are also associated with more severe attacks such as data reconstruction.
  • Dataset contamination: Prior contamination methods use n-gram or token-overlap criteria but generally require access to retraining corpora.This access is largely unavailable for recent model releases.
  • Dataset contamination: Corpus-free approaches prompt models to generate dataset examples and use verbatim memorization as evidence of contamination.The reviewed work includes methods that provide a dataset name and split, with later extensions targeting more memorized instances.

9 CONCLUSION

The paper introduces WIKIMIA and MIN-K% PROB for pretraining-data detection and evaluates the approach on contamination, book detection, and unlearning. Its experiments support the method’s effectiveness across these applications and indicate possible copyrighted-book training in GPT-3 models.

  • Contributions: WIKIMIA provides a pretraining-data detection dataset, while MIN-K% PROB uses fewer very-low-probability outlier tokens as its detection intuition.The method is evaluated in dataset contamination and published-book detection case studies.
  • Findings: Dataset-contamination results align with theoretical predictions involving dataset size, example frequency, and learning rate.The conclusion summarizes the empirical relationship between these training factors and detection difficulty.
  • Findings: Book-detection experiments provide strong evidence that GPT-3 models may have been trained on copyrighted books.This is presented as the paper’s most striking real-world finding.

A ADDITIONAL RESULTS

The additional results cover benchmark comparisons, downstream contamination detection, memorization effects under different learning rates, and similarity-score evaluation.

  • TPR@5%FPR compares MIN-K% PROB with baselines for detecting pretraining examples on WIKIMIA.The table distinguishes original and paraphrase settings and marks the best score in each column.
  • TPR @ FPR=5% evaluates reference-based and reference-free methods for detecting contaminant downstream examples.The table highlights the best reference-free TPR in each column.
  • 0.04 versus 0.11 is the difference in average classification accuracy between contaminant and non-contaminant examples at learning rates of 1 × 10−5 and 1 × 10−4, respectively.The larger difference indicates more pronounced memorization than generalization at the higher learning rate.
  • Similarity scores from SimCSE and GPT-4 accompany examples of ground-truth and LLaMA2-7B-WhoIsHarryPotter completions.The examples are presented in Table 10 for comparison with the model completions.

B DETAILS OF WIKIMIA

WIKIMIA uses time-stamped Wikipedia events to provide accurate, broadly applicable, and continually refreshed evaluation data for pretraining-data detection.

  • Data properties: The benchmark is designed to evaluate pretraining-data detection methods on newly released models.Its stated properties make it suitable for repeated evaluation across models.
  • Data properties: WIKIMIA uses events after LM pretraining as non-member data, guaranteeing that these examples were absent during pretraining.Time-sensitive events avoid the ambiguity of newer pages that revise older content already seen during training.
  • Data properties: WIKIMIA targets models pretrained on Wikipedia, including OPT, LLaMA, GPT-Neo, and Pythia.This design supports evaluation across multiple models that use Wikipedia as a pretraining source.
  • Data properties: WIKIMIA is continually updated with recent Wikipedia events so its non-member data remains current for newly introduced pretrained models.The benchmark is intended for ongoing MIA evaluation as new models are released.

C DETAILS OF MIN-K% PROB

MIN-K% PROB scores the lowest-probability tokens in an input sequence and classifies the sequence as member or non-member using a threshold.

  • Algorithm 1 Pretraining Data Detection: The algorithm computes −log p(x_i|x_1, ..., x_{i−1}) for every token in the input sequence.Each token is scored from its conditional probability given preceding tokens.
  • Algorithm 1 Pretraining Data Detection: MIN-K% PROB averages the negative log probabilities of the top k% lowest-probability tokens.These selected tokens form Min-k%(x), which supplies the sequence score.
  • Algorithm 1 Pretraining Data Detection: If MIN-K% PROB(x) exceeds ϵ, the sequence is classified as non-member; otherwise, it is classified as member.The decision rule compares the score with a specified threshold.
Loading 2310.16789v3…