PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Memorization: Violating Privacy Via Inference with Large Language Models

Robin Staab, Mark Vero, Mislav Balunović, Martin Vechev

arXiv:2310.07298v2cs.AIcs.LG

TL;DR

Existing LLM privacy research focuses mainly on memorized-data extraction, leaving open whether models can infer personal attributes from text. This study evaluates that risk using labeled Reddit profiles and examines privacy-invasive chatbots and current mitigations, finding accurate, scalable inference while anonymization and alignment remain insufficient.

  • Problem

    Privacy research has mainly examined extraction of memorized training data, leaving the inference of personal attributes from input text insufficiently studied.

  • Method

    The study evaluates nine state-of-the-art LLMs on eight personal attributes using manually labeled profiles from the PersonalReddit dataset, and examines adversarial chatbots, anonymization, and alignment.

  • Results

    LLMs achieve approximately 85% top-1 and 95.2% top-3 accuracy, with 100× lower cost and 240× lower time than human labelers, while existing anonymization and alignment remain insufficient.

  • Takeaways & Limitations

    Current LLMs can enable inference-based privacy violations at scale, including through privacy-invasive chatbots that elicit sensitive information through seemingly benign interaction.

  • Takeaways & Limitations

    Author-profiling evaluation is constrained by the scarcity of English datasets with ground-truth labels spanning broad personal-data attributes.

Abstract

from arXiv · show

Current privacy research on large language models (LLMs) primarily focuses on the issue of extracting memorized training data. At the same time, models' inference capabilities have increased drastically. This raises the key question of whether current LLMs could violate individuals' privacy by inferring personal attributes from text given at inference time. In this work, we present the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We construct a dataset consisting of real Reddit profiles, and show that current LLMs can infer a wide range of personal attributes (e.g., location, income, sex), achieving up to $85\%$ top-1 and $95\%$ top-3 accuracy at a fraction of the cost ($100\times$) and time ($240\times$) required by humans. As people increasingly interact with LLM-powered chatbots across all aspects of life, we also explore the emerging threat of privacy-invasive chatbots trying to extract personal information through seemingly benign questions. Finally, we show that common mitigations, i.e., text anonymization and model alignment, are currently ineffective at protecting user privacy against LLM inference. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale. In the absence of working defenses, we advocate for a broader discussion around LLM privacy implications beyond memorization, striving for a wider privacy protection.

1 INTRODUCTION

The paper broadens LLM privacy concerns beyond memorization by studying inference-based extraction of personal attributes from user-written text. It reports accurate, scalable inference, privacy-invasive chatbot interactions, and ineffective current mitigations.

  • This Work: Privacy Violations through LLM Inference: LLMs can infer private attributes from large collections of unstructured online text, lowering the cost of privacy-infringing profiling at scale.The paper frames this as a privacy risk distinct from extracting memorized training data.
  • This Work: Privacy Violations through LLM Inference: A Melbourne location can be inferred from an apparently ordinary reference to a “hook turn,” illustrating how models exploit subtle linguistic cues.The user did not intend to reveal their location, but GPT-4 correctly deduced Melbourne.
  • This Work: Privacy Violations through LLM Inference: ∼85% top-1 and ∼95.2% top-3 accuracy were achieved across 8 personal attributes on real-world data using 9 state-of-the-art LLMs.The models required 100× less financial and 240× lower time investment than human labelers.
  • Emerging Frontiers: Malicious chatbots can steer seemingly benign conversations toward responses containing enough information to infer private and potentially sensitive attributes.This active threat extends passive analysis of already-existing text by allowing the model to influence the interaction.
  • Potential Mitigations: State-of-the-art text anonymization remains insufficient because LLMs can use subtle regional language and contextual clues that anonymizers do not remove.The paper advocates stronger anonymization methods as LLM capabilities increase.
  • Potential Mitigations: Most current alignment methods do not filter privacy-invasive prompts, leaving privacy protection as an open research direction.Existing alignment research has primarily targeted unsafe, offensive, or biased generations.
  • Main contributions: The paper formalizes inference-based privacy threats, evaluates LLM attribute inference, and releases code, prompts, synthetic chatlogs, and 525 human-labeled synthetic examples.The contributions include a formalization, experimental evaluation, and research resources.

2 RELATED WORK

Related work has largely studied memorization and narrow author-profiling tasks, while legal frameworks define personal data broadly. The paper positions LLM inference as a broader privacy threat across data presented at inference time.

  • Privacy Leakage in LLMs: Memorization reproduces training-data sequences, whereas inference-based privacy violations can occur on any data presented to an LLM.The paper argues that memorization therefore does not capture the full extent of LLM privacy risks.
  • Risks of Large Language Models: Current LLM risk research and provider mitigations primarily target harmful, biased, or toxic generations rather than privacy-invasive inferences.Model alignment is described as the most common provider-side mitigation.
  • Personal data and PII: GDPR defines personal data as information relating to an identified or identifiable natural person, explicitly including location and economic, cultural, or social identity.U.S. PII definitions are described as less rigorous but similarly acknowledge sensitive data such as race, sexual orientation, and religion.
  • Author Profiling: Traditional author profiling extracts attributes from text but commonly focuses on gender and age using specific features and limited domains such as Twitter.A lack of labeled datasets has slowed progress in the field.

3 THREAT MODELS

The paper formalizes two privacy threats: inferring attributes from unstructured text and actively steering chatbot conversations to elicit private information. It evaluates these threats using pretrained LLMs and finds that such attacks are practical, scalable, and achievable with current models.

  • 3.1 FREE TEXT INFERENCE: A1 adversaries infer personal attributes from datasets of individuals’ unstructured texts, such as scraped online posts or social-media content.
  • 3.1 FREE TEXT INFERENCE: Given a user-text pair, A1 formats the text into a prompt, and the pretrained model returns inferred attribute–value tuples with optional reasoning.
  • 3.1 FREE TEXT INFERENCE: Pretrained off-the-shelf LLMs reduce adversaries’ training costs and entry barriers, enabling inference attacks to scale.
  • 3.1 FREE TEXT INFERENCE: LLMs approach or sometimes surpass human labelers on real-world free-text inference, using subtle language and contextual cues that can persist after anonymization.
  • 3.2 ADVERSARIAL INTERACTION: A2 adversaries deploy seemingly benign chatbots that steer users toward responses revealing information sufficient to infer private attributes.
  • 3.2 ADVERSARIAL INTERACTION: In each interaction round, the model refines prior information while keeping the hidden inference task undisclosed to the user.

4 A DATASET FOR LLM-BASED AUTHOR PROFILING

The authors introduce PersonalReddit to evaluate LLM-based author profiling on realistic online language and a broad set of labeled personal attributes. The dataset addresses limitations of existing profiling resources, which typically lack ground-truth labels or cover only a few attributes.

  • Dataset limitations: Existing author-profiling datasets are scarce, and common privacy datasets such as Enron lack ground-truth attribute labels.
  • Key requirements: The dataset requirements prioritize realistic online language and diverse personal attributes associated with each text.
  • The PersonalReddit Dataset: PersonalReddit contains 520 randomly sampled public Reddit profiles and 5,814 comments collected between 2012 and early 2016.
  • The PersonalReddit Dataset: The dataset covers eight attribute categories: age, education, sex, occupation, relationship state, location, place of birth, and income.
  • The PersonalReddit Dataset: PersonalReddit contains 1,184 total attribute labels distributed across hardness scores.

5 EVALUATION OF PRIVACY VIOLATING LLM INFERENCES

The evaluation tests nine state-of-the-art LLMs on PersonalReddit and finds strong attribute-inference performance, including in adversarial chatbot interactions. Performance varies by model size and attribute difficulty, while simulated chatbots also recover private attributes from seemingly benign conversations.

  • Free Text Inference: 85.5% top-1 accuracy was achieved by GPT-4 across PersonalReddit attributes, rising to 95.2% for top-3 predictions.GPT-4 performed best across the evaluated models.
  • Free Text Inference: 51% accuracy for Llama-2 7B increased to 66% for Llama-2 70B, with the size-related trend persisting across model families.The comparison assumes common estimates of model sizes.
  • Individual Attributes: At least 60% accuracy was obtained for every GPT-4 attribute, with nearly 97% for gender, 92% for place of birth, and 86% for location.Income had the lowest performance and only 40 samples; top-2 accuracy for income reached 87%.
  • Hardness: Accuracy decreased clearly as hardness scores increased across representative models from each family.Figure 5 reports accuracies in percent for each hardness level.
  • Adversarial Interaction: 59.2% top-1 accuracy was achieved by the adversary in 224 simulated interactions across 20 profiles, including 60.3% for location, 49.6% for age, and 67.9% for sex.The simulated results were described as similar to GPT-4’s PersonalReddit performance.

6 EVALUATION OF CURRENT MITIGATIONS

The paper evaluates client-side anonymization and provider-side alignment as mitigations against privacy-invasive LLM inference. Both are currently insufficient: anonymization leaves contextual clues usable for inference, while models reject only a small fraction of privacy-invasive prompts.

  • Client-Side Anonymization: An industry-standard AzureLanguageService anonymizer removes detected locations, addresses, persons, organizations, events, dates, ages, numbers, and currencies, but evaluation covers only location, age, occupation, place of birth, and income.The tool replaces detected entities with '*', using a certainty threshold higher than 0.4.
  • Client-Side Anonymization: GPT-4 location accuracy drops from ∼86% to still ∼55% after anonymization, despite explicit removal of location mentions.Similar behavior occurs for age, income, and place of birth.
  • Client-Side Anonymization: Anonymization effectiveness declines with inference hardness: accuracy decreases by 41.1% at hardness 1 but only 7% at hardness 5.The techniques primarily work when personal attributes are directly stated.
  • Client-Side Anonymization: Personal attributes often remain inferable from contextual language, such as region-specific slang or phrases, that current anonymization tools do not remove.The findings motivate stronger anonymizers capable of keeping up with LLM capabilities.
  • Provider-Side Alignment: Current models are not aligned against privacy-invasive prompts, and across providers only a small fraction of requests are refused.Google’s PaLM-2 models reject 10.7% of prompts, although some refusals may reflect sensitive-topic safety filters.

7 CONCLUSION

The paper concludes that pretrained LLMs can infer personal attributes from text at near-human performance and at a fraction of human cost and time. Existing anonymization and alignment mitigations are insufficient, while privacy-invasive chatbots represent an emerging threat.

  • Conclusion: The study finds that pretrained LLMs achieve near-human performance when inferring a wide range of personal attributes from text at a fraction of human cost and time.The authors describe this as making inference-based privacy violations at scale possible for the first time.
  • Conclusion: Existing anonymization and model-alignment mitigations are insufficient to adequately protect user privacy against automated LLM inference.The conclusion calls for improvements in both approaches.
  • Conclusion: Privacy-invasive chatbots constitute an emerging threat because they can extract personal information through seemingly benign interactions.The paper introduces and formalizes this threat.
  • Conclusion: The findings broaden discussion of LLM privacy beyond memorized training data to inference-based privacy implications.The authors frame this as opening a new discussion around wider privacy protection.

ETHICS STATEMENT

The ethics statement describes disclosure to model providers and safeguards for the PersonalReddit dataset. The authors emphasize that the work exposes privacy risks while avoiding publication of the sensitive dataset itself.

  • Responsible Disclosure: The authors contacted all model providers before publication to inform them about the privacy-inference issue.They also gave providers access to all their data, resulting in an active discussion about privacy-invasive LLM inferences.
  • Dataset Protection: The PersonalReddit dataset was protected by avoiding contract-worker labeling and withholding the resulting dataset from publication.The authors instead offer synthetically created samples for non-invasive further research.
  • Dataset Protection: All examples shown in the paper are synthetic, while their core content remains closely aligned with PersonalReddit samples.This choice was intended to protect individuals’ privacy without misleading readers.
  • Implications: The authors state that current anonymization methods do not protect privacy as effectively as hoped, although the underlying profiling actions were already possible before this work.They present raising awareness as a first step toward mitigating larger privacy impacts.

A DATASET STATISTICS

PersonalReddit contains 520 manually labeled Reddit profiles and 5,814 comments collected from 2012 to early 2016. Labelers also recorded certainty and hardness, and Section 5 plots use labels with certainty of at least 3.

  • Dataset Composition: The PersonalReddit dataset contains 520 manually labeled profiles comprising 5,814 comments from 2012 to early 2016.The raw data came from the PushShift Dataset.
  • Annotation: For each label, human annotators provided certainty and hardness scores on a 1-to-5 scale.The scale ranges from 1, very low, to 5, very high.
  • Annotation: Plots in Section 5 are restricted to labels with certainty of at least 3.

A.1 HARDNESS AND CERTAINTY DISTRIBUTIONS

The PersonalReddit labels were generally assigned with high certainty, while most labels had low hardness. The section also describes the profile and example distributions used to characterize the dataset.

  • Hardness and certainty: 9.9% of labels had certainty below 3, indicating that labelers were generally quite certain in their assignments.
  • Hardness and certainty: Most labels had hardness 2, with the distribution decreasing at higher hardness levels.
  • Hardness and certainty: The dataset includes marginal and joint visualizations of hardness and certainty for individual attributes and across all attributes.
  • Profile overview: Profiles contain individual comments, with a strong peak among profiles containing 0–5 comments and most average profile lengths between 0 and 4000 characters.
  • Qualitative examples: Qualitative examples cover hardness levels from 1, very easy, to 5, very hard; labelers could use online search for levels 4 and 5.

A.5 CROSS-LABELING

The cross-labeling analysis assessed agreement between labelers on a subset of PersonalReddit attributes and found high agreement for labels assigned with sufficient certainty.

  • Cross-labeling: More than 90% of labels were agreed upon when both labelers reported certainty of at least 3.
  • Cross-labeling: Among 246 cross-labeled attributes, 222 had agreement between the two labelers at certainty of at least 3.
  • Cross-labeling: Only 7 of the 24 non-aligned examples showed strong disagreement, while the remainder involved less precise or closely related labels.

B DECONTAMINATION STUDY

The decontamination study tested whether PersonalReddit comments had been memorized by the evaluated models. Across the reported metrics, the models showed no evidence of memorizing the comments.

  • Study design: The study selected 720 PersonalReddit comments of at least 100 GPT-4 tokens and randomly split each into a prefix and suffix.
  • Study design: The study evaluated generated continuations against real suffixes using string similarity, BLEU-4, token equality, longest prefix match, and longest common substring.
  • Findings: Across all metrics, the tested models had not memorized the PersonalReddit comments.
  • Findings: Among continuations with string similarity above 0.6, manual inspection found common jokes, URLs, a mathematical computation, a law paragraph, and an online meme.

C EVALUATION

The evaluation standardized model outputs and matching rules across attribute types, then reported top-k and attribute-level accuracy. Allowing multiple guesses substantially increased accuracy, with GPT-4 reaching 95.2% top-3 accuracy.

  • Evaluation procedure: Models were prompted to produce guesses in a structured format, but outputs from most models required GPT-4 reformatting and sometimes manual extraction.
  • Evaluation procedure: Categorical predictions were matched using Jaro-Winkler distance, while non-categorical predictions required a similarity of at least 0.75.
  • Evaluation procedure: Age predictions were counted correct within a 5-year radius, with range predictions evaluated using symmetric overlap or containment rules.
  • Top-k accuracies: 95.2% top-3 accuracy was achieved by GPT-4, nearly matching the human target labels.
  • Less precise answers: Less precise but not incorrect predictions were counted separately from incorrect answers, such as predicting Ohio for the ground-truth location Cleveland, Ohio.
  • Model performances across attributes: Model accuracies were reported separately for every model and personal attribute.

D ACS EXPERIMENTS

GPT-4 outperforms a majority-class baseline and nearly matches finetuned XGB models on several ACSIncome attributes, despite receiving no task-specific finetuning.

  • GPT-4 noticeably outperforms the majority-class baseline across all ACSIncome experiments.
  • GPT-4 almost matches XGB performance for place-of-birth, income, and gender without finetuning on the approximately 100k-example training set.
  • The experiment clusters several target categories, including education and race categories, before prediction.

E PAN DATASETS

The PAN comparison evaluates zero-shot GPT-4 gender classification on profiles of 100 tweets, using a sampled training subset because the competition test set was unavailable.

  • Each of the 3000 PAN profiles contains 100 tweets labeled with the author’s gender, with balanced gender labels.
  • GPT-4 was evaluated in a zero-shot setting and was not trained on the sampled PAN data.
  • The comparison sampled training data to match the final test-set size because the competition test set was inaccessible.

F SYNTHETIC EXAMPLES

Because the main PersonalReddit dataset is not released for ethical reasons, the authors create synthetic examples to support testing and qualitative reproducibility, then evaluate GPT-4 across difficulty levels.

  • 525 synthetic examples were created to enable testing of privacy-inference capabilities without releasing the PersonalReddit dataset.
  • The synthetic data generation uses adversarial investigator and user prompts spanning eight features and five hardness levels.
  • GPT-4 achieves 73.7% overall accuracy on the synthetic examples, with accuracy ranging from 94.7% to 64.7% across hardness levels.
  • The anonymization procedure uses AzureLanguageServices to remove explicitly recognized personal-attribute entities at a 0.4 recognition threshold.

H ACHIEVABLE SPEEDUP

The paper estimates substantial GPT-4 speed and cost advantages over human labeling and describes adversarial chatbot prompts designed to elicit verbose answers containing inferable personal information.

  • Achievable speedup: GPT-4 takes around 5–20 seconds per profile, whereas human labelers take a few minutes for an average-length profile.
  • Achievable speedup: Human labeling of PersonalReddit required roughly 112 person-hours, while GPT-4 processing used parallel workers to reduce runtime.
  • Achievable speedup: The estimated human-labeling cost was roughly 2250 USD, approximately 100× the GPT-4 cost.
  • Adversarial interaction: The adversarial chatbot prompts instruct an investigator to conceal its inference goal, ask subtle follow-up questions, and elicit verbose user responses.
  • Adversarial interaction: The investigator is instructed to return three ordered guesses with reasoning about the user’s inferred attribute.

I.1 REDDIT EXPERIMENTS

The Reddit experiments prompt LLMs to infer target personal attributes from an author's public comments, including increasingly precise location, age, and sex. The examples show models deriving location clues from landmarks and geography while using weaker linguistic and demographic stereotypes for age and sex.

  • Experiment setup: The PersonalReddit setup asks models to infer specified attributes from one author's public Reddit comments and provide three guesses for each value.The prompt requests step-by-step reasoning over textual information and language use before producing top-three guesses.
  • Attribute targets: Location inference seeks the most exact deducible residence, potentially narrowing from country to city, district, or street.The operational definition uses San Francisco as an example and prioritizes more precise geographic information when available.
  • Location inference: Landmarks and geographic references support specific city predictions, including Auckland from the Waitakere Ranges and Munich from the Isar River and Frauenkirche.The Auckland example produces top guesses of Auckland, Wellington, and Christchurch; the Munich example produces Munich, Stuttgart, and Frankfurt.
  • Age inference: Age predictions rely on life-stage, platform-use, education, and family-history cues but remain explicitly uncertain in the examples.One example gives guesses of 35, 40, and 45 while stating that the specific age remains unclear; another places the user in a university-student or young-professional range.
  • Sex inference: Sex predictions frequently infer male identity from language, hobbies, and topic associations, while some examples acknowledge that these are stereotypical rather than definitive clues.The examples cite terms such as “mate,” “bloke,” and “oi mate,” alongside scientific interests, coding, and gardening language.
Loading 2310.07298v2…