PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Leakage of Personally Identifiable Information in Language Models

Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, Santiago Zanella-Béguelin

arXiv:2302.00539v4cs.LG

TL;DR

PII leakage from language models is poorly quantified, despite imperfect scrubbing and uncertainty about how DP protects PII. The paper formalizes three black-box attack types and evaluates them on defended and undefended GPT-2 models. DP substantially reduces leakage but does not eliminate it, motivating complementary privacy measures.

  • Problem

    PII leakage in language models lacks end-to-end quantitative metrics, while the protection provided by scrubbing and differential privacy remains unclear.

  • Method

    The paper defines extraction, reconstruction, and inference games and evaluates corresponding black-box attacks on GPT-2 models across law, health-care, and e-mail datasets.

  • Results

    DP significantly bounds PII reconstruction leakage, but PII extraction and inference remain possible, reaching up to 10% inference accuracy with 100 candidates.

  • Takeaways & Limitations

    DP is useful but insufficient alone; PII scrubbing remains necessary, and less aggressive DP-informed scrubbing could preserve utility while maintaining protection.

  • Takeaways & Limitations

    The study uses the same NER model for scrubbing and leakage measurement because extensive annotated PII benchmarks are unavailable, and broader validation is needed.

Abstract

from arXiv · show

Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage. Scrubbing techniques reduce but do not prevent the risk of PII leakage: in practice scrubbing is imperfect and must balance the trade-off between minimizing disclosure and preserving the utility of the dataset. On the other hand, it is unclear to which extent algorithmic defenses such as differential privacy, designed to guarantee sentence- or user-level privacy, prevent PII disclosure. In this work, we introduce rigorous game-based definitions for three types of PII leakage via black-box extraction, inference, and reconstruction attacks with only API access to an LM. We empirically evaluate the attacks against GPT-2 models fine-tuned with and without defenses in three domains: case law, health care, and e-mails. Our main contributions are (i) novel attacks that can extract up to 10$\times$ more PII sequences than existing attacks, (ii) showing that sentence-level differential privacy reduces the risk of PII disclosure but still leaks about 3% of PII sequences, and (iii) a subtle connection between record-level membership inference and PII reconstruction. Code to reproduce all experiments in the paper is available at https://github.com/microsoft/analysing_pii_leakage.

I. INTRODUCTION

PII leakage from language models remains insufficiently understood because scrubbing is imperfect and privacy defenses create privacy/utility trade-offs. This work formalizes three black-box PII threats, develops measurable attacks, and evaluates defenses across multiple domains.

  • Motivation: Existing work does not distinguish public from private memorized information, leaving comprehensive PII-memorization risk insufficiently studied.Common duplicated phrases may not constitute privacy violations, whereas leaked PII can re-identify individuals.
  • Defenses: Scrubbing and differential privacy can reduce model utility, while the contribution of record- or user-level DP to PII protection remains unclear.Figure 2 compares undefended, scrubbed, DP, combined, and masked-output models using perplexity as the utility measure.
  • Threat Models: The paper defines extraction, reconstruction, and inference as distinct PII-leakage threats with rigorous game-based formulations.Extraction uses no dataset knowledge; reconstruction uses contextual knowledge; inference additionally uses candidate PII values.
  • Motivation: PII leakage can expose sensitive information even when an attacker cannot reconstruct its surrounding context.The paper distinguishes direct re-identification from leakage that associates PII with a person or context.
  • Results: The proposed attacks achieve approximately twice the precision of related work and can reconstruct up to 10× more PII sequences by leveraging masked-query suffixes and public masked LMs.The attacks are evaluated on GPT-2 variants fine-tuned on law, health-care, and e-mail data.
  • Results: Differential privacy substantially limits PII leakage but does not eliminate extraction, reconstruction, or inference risks.The paper presents DP as partially effective while retaining residual disclosure and motivates defenses that account for privacy/utility trade-offs.

II. BACKGROUND & PROBLEM

The paper introduces language-modeling foundations, privacy mitigations, and a problem setting for measuring PII leakage. It highlights that DP’s data-independent guarantees may not capture sensitive information shared among groups of unknown size.

  • Language Modeling: Generative LMs model the conditional probability of each token given its preceding tokens and generate text iteratively from those probabilities.Autoregressive models condition on prefixes, while masked LMs can also use a query’s suffix.
  • Language Modeling: Training can encourage memorization, although practical models memorize only a fraction of the training data and must generalize beyond it.Perplexity on unseen sentences measures utility, with lower perplexity indicating higher utility.
  • Differential Privacy: Differential privacy provides worst-case guarantees defined over adjacent datasets, whose adjacency may differ by one record or one user.Its guarantees are independent of the underlying data distribution and include properties such as post-processing robustness and composition.
  • Differential Privacy: DP’s data independence is limited when sensitive content is shared within groups of users whose size is unknown.In such settings, contextual sensitivity may not be represented by a pairwise dataset-adjacency relation.

C. PII and NER

The paper treats PII as information that can directly or indirectly re-identify individuals and examines how NER-based scrubbing handles it. Its pipeline combines data curation, algorithmic defenses, and black-box deployment while highlighting tensions between privacy protection and utility.

  • PII: PII directly re-identifies individuals alone, whereas quasi-identifiers require aggregation with other attributes.Names, phone numbers, and addresses are direct identifiers; gender and physical appearance descriptions are quasi-identifiers.
  • NER: NER tagging is difficult because PII definitions vary with time and surrounding context.NER also requires domain-specific labeled data and can miss complex PII requiring natural-language understanding.
  • PII Scrubbing: Scrubbing replaces NER-detected PII with [MASK] tokens, while weaker variants preserve entity labels or repeated pseudonyms.The paper’s scrubber maximizes privacy at the expense of some utility.
  • Training Pipeline: The considered training pipeline applies curation first, algorithmic defenses second, and exposes only next-token prediction through a black-box API.Model parameters and intermediate features remain hidden from the attacker.
  • Research Questions: The paper studies whether memorized information is PII, whether defenses prevent leakage, and how defenses trade privacy against utility.This includes whether less aggressive scrubbing could be used with differential privacy.
  • Privacy Trade-offs: Differential privacy can protect against singling out contributors but may not protect information about whom the content concerns.Repeated sensitive information across users can remain exposed even when contributor authorship is protected.

E. Threat Model

The threat model gives an attacker black-box API access and distinguishes extraction, reconstruction, and inference by the attacker’s knowledge. The paper defines extractability and estimates it with sampled generations, while noting that grammar can make naive sampling inefficient.

  • Adversary Capabilities: The adversary queries the model’s full next-token probability vector through a black-box API.Query costs limit how many attacks can be mounted in practice.
  • Threat Models: Extraction targets any training-seen PII without dataset knowledge, reconstruction uses masked context, and inference additionally supplies candidate PII sequences.Reconstruction and inference aim to associate PII with context, unlike extraction’s broader goal.
  • Game-Based Definitions: Attack success is defined through recall, while adversarial advantage compares success with the best adversary lacking model-oracle access.The game-based formulation parameterizes the experiment by the training algorithm, data distribution, and dataset size.
  • Extractability: PII extractability is the expected probability that a target PII sequence appears in model-generated samples.Frequently occurring PII is expected to have higher generation likelihood, while unlikely prompts can keep memorized PII difficult to extract.
  • Estimation: The formal extractability sum over all possible sequences is intractable, so a baseline estimates it by counting target occurrences in generated sentences.The paper also proposes a sample-efficient estimator using grammatical rules and interchangeability within PII classes.
  • Estimation: Naive sampling may require many generations because grammar restricts where PII such as proper names can appear.This creates a practical limitation for obtaining a good lower bound on extractability.

B. PII Reconstruction

PII reconstruction asks an attacker to recover masked information from its surrounding context. The proposed attack approximates an intractable constrained search by generating candidates and ranking them using the model’s perplexity on the complete sequence.

  • Threat Model: In reconstruction, the attacker receives a scrubbed sentence’s prefix and suffix and must recover a selected masked PII sequence.The reconstruction game samples a training sentence containing PII and randomly selects the target.
  • Attack Procedure: The attack fills residual masks with a public masked language model before searching for the target PII.This preprocessing step handles masked pieces unrelated to the selected target.
  • Candidate Generation: Without candidate PII, the attack generates top-k samples from the target model and gathers generated PII pieces as candidates.With candidates, the same framework can rank a supplied set rather than generate it.
  • Candidate Ranking: Candidates are ranked by the target model’s perplexity on the entire sequence, and the best candidate is returned.Using both prefix and suffix improves on the prior TAB attack, which greedily decodes from the prefix alone.
  • Computational Constraint: The exact reconstruction objective is intractable because the attacker does not know the target PII’s token length and must search valid PII sequences.The proposed procedure therefore approximates constrained beam search.

C. PII Inference

PII inference is reconstruction with an informed adversary who knows a candidate set containing the target. The paper encodes this setting as a game and evaluates a constrained reconstruction attack against it.

  • Threat Model: PII inference assumes the adversary knows candidate PII sequences and must identify the correct one.The candidate set is assumed to contain the target sequence.
  • Game Definition: The inference game samples a sentence containing PII and selects a PII sequence from it as the target.The target distribution is defined by sampling from the PII-bearing sentence population and choosing uniformly among its PII sequences.
  • Attack Procedure: The attack incorporates the candidate set into the reconstruction procedure to approximate the inference objective.This constrains the search to the supplied PII candidates.
  • Empirical Observation: The attacker can infer PII in differentially private models trained with ε = 8.The paper reports that needing only to identify the correct candidate makes the attacker significantly more powerful.

D. Baseline Leakage

The evaluation addresses baseline leakage by accounting for PII already reproducible from the public pretrained model and studying realistic datasets across multiple domains.

  • Baseline PII is excluded using samples generated from a public model before attributing remaining leakage to private-data exposure.The broader setup also compares models trained with and without defenses.
  • Baseline leakage is difficult to handle without natural-language understanding and real-world context.This can under-count sensitive disclosures involving public figures or naming collisions.
  • The evaluation covers law cases, corporate e-mails, and healthcare-facility reviews, using realistic data from expert- and user-generated sources.The private data is split into equally large training and validation sets plus a smaller test set.
  • The study scrubs 21 PII entity classes with two NER-based taggers before evaluating leakage.The scrubber combines Flair taggers with Presidio’s default spaCy-based tagger.

C. Language Models

The paper evaluates GPT-2 models of four sizes under multiple defenses, measuring utility, membership inference, and PII leakage across three domains. Results show that duplication and sampling affect extraction, while DP reduces but does not eliminate leakage.

  • Language Models: The experiments compare undefended, DP, scrubbed, and jointly DP-and-scrubbed GPT-2 models.GPT-2 Small, Medium, Large, and XL contain 124m, 355m, 774m, and 1 557m parameters, respectively.
  • Metrics: The evaluation reports test perplexity, sentence-level membership-inference ROC AUC, PII extraction precision and recall, and reconstruction or inference top-1 accuracy.Extraction recall measures PII at risk, while precision measures confidence that generated PII appears in training data.
  • Evaluation Setup: 4m sampled tokens are generated through 15k queries of 256-token sequences using top-k sampling with k = 40.Baseline leakage is removed using 13m tokens generated by a public model across 50k queries.
  • Model Size: 23% recall and 30% precision are observed for GPT-2-Large on ECHR, while GPT-2-Small recalls about 9% at roughly 25% precision.Precision and recall decrease with model size in this comparison.
  • Risk Factors: PII duplication has a linear relationship with leakage frequency, whereas DP models leak about an order of magnitude fewer 3–6-token PII sequences than undefended models.Long PII sequences are not leaked in DP models, and single-occurrence PII is observed only in undefended models.
  • Risk Factors: Recall rises to 23% as sampling increases to 4m tokens, while precision falls from 50% at 500k tokens to 30%.Larger samples generate more PII but at lower precision.
  • Risk Factors: The estimated extractability metric correlates with observed leakage but produces false-negative outliers.It enables point-wise assessment without exhaustively sampling massive generated datasets.
  • PII Classes: DP reduces extraction of law cases to 2.8% and organizations to 4.1%, while no real Enron phone numbers are observed in the DP model.E-mail addresses still leak in Enron, typically correlated with a person’s name.

F. PII Reconstruction

The paper’s reconstruction attack uses masked queries and candidate sampling to recover PII from context. It substantially outperforms TAB on undefended models, while DP reduces reconstruction to at most 1% and larger models remain more vulnerable.

  • PII Reconstruction: The reconstruction attack outperforms TAB by up to 10× on undefended models, including GPT-2-Medium on Enron.It samples 64 candidates and decodes with top-k sampling using k = 40.
  • Model Size: On ECHR with GPT-2-Large, the attack reconstructs 18.27% of PII versus at least 5.81% for TAB.The attack improves over TAB by at least 2.5× on ECHR, 7.5× on Enron, and about 3× on Yelp-Health in the reported comparisons.
  • Defenses: Reconstruction leakage is at most 1% in DP models, although the attack still improves on the baseline in every case.The authors consider this leakage too small for a practical attack.
  • Context Size: On Enron, suffix information strengthens reconstruction because TAB cannot use the e-mail body when PII follows a short prefix.When the correct PII is among candidates, it is predicted correctly about 70% of the time; candidate sampling remains the main limitation.
  • PII Inference: PII inference uses an anonymized training dataset and candidate PII lists, reaching 70% accuracy on ECHR, 50% on Enron, and 28% on Yelp-Health without defenses.The higher leakage on ECHR and Enron is associated with repeated PII in similarly structured sentences.

H. Membership Inference and PII Leakage

The paper links sentence-level membership inference to PII reconstruction and evaluates how scrubbing and differential privacy affect privacy leakage and utility. It finds that defenses reduce leakage but leave practical risks and trade-offs.

  • AUC is 0.96 for undefended, 0.82 for scrubbed, and 0.505 for DP & scrubbed models in the sentence-level membership-inference attack.Scrubbing mitigates membership inference, but differential privacy is substantially more effective in this comparison.
  • PII inference can be transformed into a membership-inference attack, while using membership inference for PII reconstruction or inference requires distinguishing sentences differing in only a few tokens.
  • Memorization scores of generated sequences are nearly zero, yet memorization correlates with the probability of correct PII reconstruction.
  • Undefended models are highly vulnerable to membership inference, extraction, reconstruction, and inference, with larger models and higher duplication increasing PII leakage.
  • Differential privacy significantly bounds PII reconstruction leakage but does not eliminate PII inference or extraction; inference reaches up to 10% accuracy given 100 candidates.
  • DP and aggressive scrubbing reduce utility, motivating defenses with better privacy/utility trade-offs and scrubbing that accounts for DP protection.
  • Extracted PII can support later association-based attacks that reveal additional private information and enable linkability attacks.
  • The evaluation is limited by scarce annotated benchmarks, reliance on the same NER model for scrubbing and measurement, and omission of high-precision/low-recall attacks.

VI. RELATED WORK

Prior work studied training-data extraction and privacy defenses, but mainly outside end-to-end fine-tuning pipelines that combine scrubbing and differential privacy. This work evaluates those trade-offs across extraction, reconstruction, and inference threats.

  • Earlier studies examined leakage of sentences, canaries, n-grams, and PII across language and classification models.
  • Related work focused mainly on leakage without mitigations, whereas this paper evaluates PII leakage and the interplay of scrubbing and differential privacy in fine-tuned language models.
  • Prior mitigation research proposed differential-private fine-tuning and selective differential privacy to reduce utility degradation.
  • This work defines metrics and game-based threats for extraction, reconstruction, and inference while studying privacy/utility trade-offs during language-model fine-tuning.

APPENDIX A DATASETS

The appendix describes three evaluated datasets, their PII characteristics, and the preprocessing and splitting choices used for experiments.

  • The datasets cover case law, corporate e-mails, and health-related reviews, enabling evaluation across three text domains.
  • ECHR: ECHR contains 118 161 records, split so each record contains a single fact.
  • ECHR: 23.75% of ECHR records contain at least one PII sequence, with 16 133 unique person entities and mean duplication rate 4.66.
  • Enron: Enron contains about 600 000 real e-mails from 158 employees, with informal language and possible typographical errors.
  • Yelp-Health: 54.55% of Yelp-Health records contain at least one PII sequence; Flair tags 17 035 pieces averaging 2.17 tokens and duplication rate 5.53.
  • Yelp-Health: Yelp-Health reviews can contain sensitive disease descriptions with timestamps and locations despite being posted publicly.

D. PII Leakage Metrics

The paper evaluates PII leakage using extractability, reconstruction, and inference metrics, while masked queries are completed iteratively by a public masked language model.

  • PII Extractability: PII extractability is measured with precision and recall, capturing the correctness and coverage of predicted PII.
  • PII Reconstruction and Inference: PII reconstruction and inference are evaluated with top-1 accuracy.
  • PII Reconstruction and Inference: For reconstruction, the attacker guesses a randomly selected masked PII within a training sequence and compares the guess with the original.
  • The attack success depends only mildly on other PII in the sampled sequence because scrubbing removes other PII and reduces contextual information.
  • Mask Filling: Masked queries with multiple PII tokens are completed left to right using the top token predicted by a public masked language model.
Loading 2302.00539v4…