PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Dataset Inference: Did you train on my dataset?

Pratyush Maini, Hengrui Jia, Nicolas Papernot, Adam Dziedzic

arXiv:2406.06443v1cs.LGcs.CLcs.CR

TL;DR

The paper questions whether individual-sequence membership inference can reliably identify LLM training data because distribution shifts confound its apparent success. It proposes dataset inference, which combines informative attacks and aggregates their scores statistically, and reports significant train-versus-validation distinctions across Pile subsets without false positives.

  • Problem

    Existing membership-inference attacks may mistake distribution shifts for individual training-set membership, complicating attribution of copyrighted LLM training data.

  • Method

    Dataset inference selectively combines membership-inference signals across a suspect dataset and applies statistical testing under a victim-provider-arbiter framework.

  • Results

    Dataset inference distinguishes Pile training and validation splits with p-values below 0.1 across subsets and produces no false positives in validation comparisons.

  • Takeaways & Limitations

    Aggregating evidence over datasets provides a more viable attribution approach for claims that an LLM trained on multiple documents from a creator.

  • Takeaways & Limitations

    The method assumes IID training and validation sets, a completely private validation set, and a victim or trusted third party able to run the inference.

Abstract

from arXiv · show

The proliferation of large language models (LLMs) in the real world has come with a rise in copyright cases against companies for training their models on unlicensed data from the internet. Recent works have presented methods to identify if individual text sequences were members of the model's training data, known as membership inference attacks (MIAs). We demonstrate that the apparent success of these MIAs is confounded by selecting non-members (text sequences not used for training) belonging to a different distribution from the members (e.g., temporally shifted recent Wikipedia articles compared with ones used to train the model). This distribution shift makes membership inference appear successful. However, most MIA methods perform no better than random guessing when discriminating between members and non-members from the same distribution (e.g., in this case, the same period of time). Even when MIAs work, we find that different MIAs succeed at inferring membership of samples from different distributions. Instead, we propose a new dataset inference method to accurately identify the datasets used to train large language models. This paradigm sits realistically in the modern-day copyright landscape, where authors claim that an LLM is trained over multiple documents (such as a book) written by them, rather than one particular paragraph. While dataset inference shares many of the challenges of membership inference, we solve it by selectively combining the MIAs that provide positive signal for a given distribution, and aggregating them to perform a statistical test on a given dataset. Our approach successfully distinguishes the train and test sets of different subsets of the Pile with statistically significant p-values < 0.1, without any false positives.

1 Introduction

The paper argues that individual-sentence membership inference is confounded by distribution shifts and introduces dataset inference as a more realistic approach to copyright attribution. Across Pile subsets, dataset inference distinguishes training from validation data with significant p-values and no false positives.

  • Motivation: Copyright and privacy concerns motivate methods that identify training data and support data accountability for creators.The paper situates the problem in legal challenges over unlicensed copyrighted and personally identifiable information in web-scale training corpora.
  • Membership Inference: Previous membership-inference successes are overly optimistic because attacks often detect temporal or other distribution shifts rather than individual sentence membership.The paper reports that no single MIA succeeds across all data distributions.
  • Dataset Inference: The paper proposes dataset inference to test whether a collection of texts, such as an author’s books, was used to train an LLM.This framing matches copyright disputes involving multiple documents attributed to one creator.
  • Results: Dataset inference achieves p-values below 0.1 across Pile subsets while producing no false positives in validation-versus-validation comparisons.The method requires only 1000 text sequences to test whether a suspect dataset trained an LLM.

2 Background and Baselines

The background distinguishes membership inference from dataset inference and reviews several attack families used as LLM baselines. These attacks classify individual sequences using loss, perturbation behavior, reference-model comparisons, or compression-based scores.

  • Membership Inference: Membership inference asks whether a particular data point appeared in a model’s training set.The paper reviews it as a privacy, contamination, and copyright-auditing problem.
  • Dataset Inference: Dataset inference tests whether a model derives from a private training dataset, extending earlier ideas from supervised and self-supervised learning to LLMs.Earlier methods rely on decision-boundary distances or differing representation distributions between training and test examples.
  • Thresholding-Based MIAs: Thresholding attacks classify samples from loss or perplexity, but predictable non-members can produce false positives.The decision rule labels a sequence as a member when its loss falls below a threshold.
  • Other Baselines: Other baselines include Min-k% Prob, DetectGPT, reference-model ratios, and zlib-based compression ratios.Min-k% Prob focuses on the least-probable tokens, while DetectGPT compares an input with externally infilled neighbors.
  • Perturbation-Based MIAs: Perturbation-based attacks compare an original sequence with modified versions, including whitespace changes, synonym substitutions, typos, and deletions.Their hypothesis is that training members have lower perplexity on the original than on perturbed text.

3 Problem Setup

The problem setup models a copyright dispute in which a creator suspects that an LLM provider trained on their private or copyrighted data. Dataset inference is designed for black-box model access and uses an arbiter to evaluate the claim.

  • Problem: The paper asks how to detect whether a given dataset was used to train an LLM as training corpora grow to trillions of tokens.The setup is motivated by the expected degradation of individual membership inference toward random guessing on very large training sets.
  • Access Assumptions: The threat model allows limited model access, including API input-output access and model loss, without exposing weights or gradients.Such black-box access reflects how users interact with commercial LLMs.
  • Actors: Dataset inference assigns roles to a victim creator, a suspect LLM provider, and an arbiter that executes the procedure during a dispute.The victim is presumed to have black-box access, while the arbiter is a trusted third party.
  • Scenario: In the motivating scenario, a writer supplies roughly 100 manuscript sequences and an arbiter tests whether the provider trained on the writer’s published work.The procedure is presented as a way to resolve suspected infringement of copyrighted manuscripts.

4 Failure of Membership Inference

The section shows that apparent membership-inference success is largely explained by distribution shifts, while performance approaches chance on IID splits and varies across datasets and random subsets.

  • WikiMIA’s pre-2023 versus post-2023 construction confounds membership with temporal shifts in writing styles and concepts.Post-2023 articles are non-members and pre-2023 articles are treated as members, so the split also changes time period.
  • AUC is close to 0.5 for Min-k% Prob on IID Wikipedia train and validation splits, unlike its roughly 0.7 AUC on WikiMIA.This indicates performance akin to random guessing in the IID setting.
  • Min-k% Prob varies between AUC 0.4 and 0.7 across random Pile train-validation subsets.The reported variance shows that results are unstable even within the same dataset family.
  • AUC values near 0.4 on Arxiv and OpenWebText2 indicate false positives, with validation examples labeled as members.
  • Across 20 Pile subsets, no single MIA works consistently: methods can achieve high AUC on one dataset and below 0.5 on another.For example, synonym substitution performs well on PhilPapers but has AUC below 0.5 on Pubmed Abstracts.
  • Sound MIA evaluation requires IID train-validation splits, repeated random splits, multiple distributions, and explicit false-positive and false-negative checks.

5 LLM Dataset Inference

LLM dataset inference aggregates membership-inference signals across datasets, learns distribution-specific weights, and statistically tests whether a suspect dataset matches the model’s training distribution. Experiments on Pythia models and PILE subsets show reliable attribution with controlled false positives, while performance depends on preprocessing, query count, model size, and duplication.

  • Procedure for the LLM Dataset Inference: Dataset inference combines MIA features, learns their distribution-specific importance with a linear regressor, and applies a t-test to suspect and validation scores.The procedure uses held-out B splits for ownership assessment and reports whether the suspect dataset was used in training.
  • Procedure for the LLM Dataset Inference: Dataset inference combines p-values from dependent tests across random splits rather than treating overlapping tests as independent.The method uses multiple random seeds and aggregates the resulting dependent-test p-values.
  • Analysis and Results with Dataset Inference: P-values below 0.1 distinguish PILE training from validation splits, while comparisons between two validation subsets produce no false positives.The experiments use 1000 data points for attribution and evaluate false positives on separate validation subsets.
  • Analysis and Results with Dataset Inference: Feature selection varies by domain: perturbation metrics help CC, whereas perplexity is needed for OWT2.The linear classifier learns which MIAs are positively correlated with each distribution.
  • Analysis and Results with Dataset Inference: Removal and normalization of outliers is more reliable than alternative preprocessing because other transformations can create false positives.The selected approach eliminates outliers and normalizes feature values before regression.
  • Analysis and Results with Dataset Inference: 1000 queried points suffice to reach p-values below 0.1 for all datasets, while more than half require only about 100 points.Query requirements vary across datasets.
  • Analysis and Results with Dataset Inference: Attribution success increases with model size and is stronger for non-deduplicated training data under a 500-query budget.Larger models concentrate p-values below 0.1, consistent with stronger memorization signals.

6 Discussions

The discussion argues that individual-example MIAs are unreliable for LLMs because apparent success can reflect distributional confounders. It presents dataset inference as a more suitable attribution approach, while noting assumptions about IID data, private validation sets, and authorized access.

  • Membership Inference for LLMs: Current LLM membership-inference attacks can perform no better than random guessing when experiments control for distributional confounders.The paper recommends IID splits, false-positive checks, varied distributions, and multiple random seeds.
  • Shift to LLM Dataset Inference: Dataset inference aggregates examples from an entity to detect training of a dataset rather than membership of an individual sentence.The paper frames this shift around real-world claims that books or other collections were used for training.
  • Limitations: Dataset inference requires IID train and validation sets, a completely private validation set, and a victim or trusted third party to run the assessment.These access and data assumptions may require a legal framework because an adversary might otherwise refuse model queries.

A Broader Impact

The paper situates LLM dataset inference in privacy and copyright concerns arising from web-scraped training data. It presents the method as statistically grounded and thoroughly evaluated.

  • Web-scraped LLM training data raises privacy concerns because it may include personally identifiable information and copyrighted material.
  • The paper presents its LLM dataset inference method as statistically grounded and thoroughly evaluated.

B Compute

The experiments required substantial GPU memory and multiple machines to aggregate membership-inference metrics through large-batch model forward passes.

  • Four NVIDIA A6000 48GB machines were used at a time to aggregate membership-inference metrics efficiently.The setup supported large-batch forward passes while loading the required models.

C Additional Experiments

Additional experiments test membership-inference consistency, feature behavior, preprocessing, and dataset-inference sensitivity to data quantity and model size. They reinforce that signals vary across datasets while dataset inference benefits from selective aggregation and larger inputs.

  • None of the membership-inference attacks consistently achieves high ROAUC across different datasets.Only a few methods meet the effectiveness standard on particular datasets, with DetectGPT performing especially well on some book datasets.
  • Most features contribute positively for some datasets but negatively for others, motivating dataset-specific aggregation rather than reliance on one feature.
  • Normalization improves the maximum p-value across datasets, while preprocessing has little effect on the median and Wikipedia curves.
  • Dataset inference is accurate for most datasets with fewer than 100 points, and larger models concentrate p-values below the significance threshold.
Loading 2406.06443v1…