Source-linked AI summary
Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, Yejin Choi
TL;DR
Interactive LLM applications require models to decide what information may be shared, with whom, and for what purpose, yet inference-time contextual privacy remains understudied. The paper introduces CONFAIDE, a multi-tier benchmark grounded in Contextual Integrity and theory of mind. GPT-4 and ChatGPT nevertheless disclose or misroute private information frequently, and prompting or chain-of-thought reasoning does not resolve the problem.
Problem
Interactive LLMs receive information from multiple sources and must reason about context-dependent sharing, a form of inference-time privacy insufficiently addressed by training-data privacy measures.
Method
The paper introduces CONFAIDE, a four-tier benchmark that evaluates LLM privacy reasoning across increasingly complex information-flow contexts involving sensitivity, actors, uses, relationships, and multiple information types.
Results
GPT-4 and ChatGPT flow information to inappropriate actors 39% and 57% of the time, respectively, in Tier 4 despite direct privacy instructions.
Takeaways & Limitations
The findings indicate that safer contextual privacy behavior requires approaches targeting reasoning and theory of mind rather than only surface-level safeguards.
Takeaways & Limitations
The paper notes that inference-time privacy concerns remain only partially explored, including possible leakage of in-context examples and other unexplored risks.
Abstract
from arXiv · showhide
The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.) introduces a new set of inference-time privacy risks: LLMs are fed different types of information from multiple sources in their inputs and are expected to reason about what to share in their outputs, for what purpose and with whom, within a given context. In this work, we draw attention to the highly critical yet overlooked notion of contextual privacy by proposing ConfAIde, a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. Our experiments show that even the most capable models such as GPT-4 and ChatGPT reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. This leakage persists even when we employ privacy-inducing prompts or chain-of-thought reasoning. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
1 INTRODUCTION
Interactive LLM assistants create inference-time privacy risks because models must reason about information flows across sources, people, purposes, and contexts. CONFAIDE tests this capability across increasingly complex contextual-integrity tiers and finds substantial divergence from human privacy expectations.
- Motivation: Existing LLM privacy research largely studies training-data memorization and leakage, leaving interactive input-to-output information flows less examined.The gap concerns settings where models receive information from multiple sources and generate context-dependent replies.
- Motivation: Contextual Integrity defines privacy as appropriate information flow within a specific social context, not solely by the information’s inherent sensitivity.A breach occurs when information flows against the norm of its context.
- Benchmark: CONFAIDE evaluates contextual privacy reasoning across tiers that increase from single-information sensitivity judgments to multiple information types flowing among multiple people.The benchmark incorporates actors, uses, and increasingly complex social contexts, including meeting summarization and action-item generation.
- Results: GPT-4’s human-model privacy correlation drops from 0.8 to 0.1 from Tier 1 to Tier 3.Higher tiers are designed to more closely mirror real-world scenarios and elicit more frequent private-information disclosure.
- Results: GPT-4 and ChatGPT flow information to inappropriate actors 39% and 57% of the time, respectively, in Tier 4 despite direct privacy instructions.In Tier 3, they reveal secrets 22% and 93% of the time, respectively.
2 BACKGROUND & RELATED WORKS
Contextual privacy evaluates whether information flows conform to norms specific to a social domain, distinguishing this perspective from context-independent training-data protections. Because secret keeping requires tracking others’ knowledge and mental states, theory of mind is central to contextual privacy reasoning.
- Contextual Integrity: Contextual integrity holds that privacy violations occur when information flows deviate from the norms of a particular social context.Relevant contexts include health, work, family, and civil and political domains.
- Contextual Integrity: Appropriate information flows depend on contextual norms, including whether information is shared appropriately, with consent, and for an intended purpose.The same information may be acceptable to share in one context but inappropriate in another.
- LLM Privacy: Differential privacy protects training records with context-independent, worst-case guarantees rather than modeling contextual information flow.It makes models trained on neighboring datasets indistinguishable at the record level.
- Theory of Mind: Theory of mind is the ability to comprehend and track others’ mental states and knowledge, making it important for reasoning about secrets and information asymmetry.Privacy decisions require understanding who can access particular information and how social norms govern sharing.
3 CONFAIDE: BENCHMARKING CONTEXTUAL PRIVACY REASONING IN LLMS
CONFAIDE benchmarks LLMs’ contextual privacy reasoning through four tiers that progress from sensitivity judgments to complex multi-party information-flow decisions. Its evaluations combine human-aligned expectations with tests of leakage, accessibility tracking, private-information sharing, and privacy–utility trade-offs.
- Benchmark Design: CONFAIDE has four tiers whose contexts become more intricate and increasingly require social reasoning such as theory of mind.The first two tiers draw on empirical human privacy-expectation research, while higher tiers introduce richer social contexts.
- Tier 1: Tier 1 asks models to rate the sensitivity of information types using four ratings and compares responses with human annotations.The prompt asks how sensitive people would consider each information type, avoiding questions framed as the model’s own opinion.
- Tier 2: Tier 2 constructs factorial vignettes from information type, actor, and use to evaluate whether information flows meet people’s privacy expectations.Models rate each scenario from strongly disagree to strongly agree using a prompt adapted from Martin and Nissenbaum’s study.
- Tier 3: Tier 3 tests nuanced three-party scenarios involving information type, relationships, and incentives through response generation and information-accessibility judgments.Additional tasks assess private-information sharing and a binary control question about whether Y may share X’s secret with Z.
- Tier 3: Tier 3 leakage is assessed by exact name matching and by testing whether a proxy model can recover the private information from the response alone.The proxy model is Llama-2-13b-chat, prompted to identify who is related to the secret.
- Tier 4: Tier 4 evaluates meeting action-item and summary generation, requiring models to exclude sensitive information while retaining vital public information.The design explicitly creates a privacy–utility trade-off across two real-world application tasks.
- Human Evaluation: Human annotations use five Mechanical Turk workers per sample across Tiers 1–3, with different rating and response-choice procedures by tier.Tier 3 compares a secret-revealing response with a generic response omitting the secret.
- Human Evaluation: 90% of Tier 3 samples preferring privacy received at least 4 of 5 votes, while only 9 of 270 scenarios received a majority vote to disclose.The nine disclosure-preferring scenarios received no more than 3 of 5 votes.
4 EXPERIMENTAL RESULTS
Across four tiers, model alignment with human privacy judgments declines as contextual complexity increases, while leakage remains substantial even under privacy-preserving instructions. GPT-4 and ChatGPT perform better than many models on some measures, but still fail to reliably control contextual information flow.
- Experimental setup: The experiments evaluate GPT-4, ChatGPT, Davinci1, Llama-2 Chat, Llama 2, and Mixtral across four increasingly complex tiers.Metrics are averaged over 10 runs, with additional breakdowns and experiments reported in the appendices.
- Alignment with human judgment: As tiers become more nuanced, agreement between human and model judgments decreases, although heavily RLHF-trained and instruction-tuned models align more closely with humans.For GPT-4, correlation drops from 0.8 in Tier 1 to 0.1 in Tier 3.
- Tier 1 and Tier 2 results: GPT-4’s contextual judgments change substantially across tiers, including treating SSN as less sensitive when shared with insurance and with a doctor in a more nuanced scenario.The reported shifts span −100 to −25 in both examples.
- Tier 3 analysis: GPT-4’s Tier 3 leakage varies by contextual factor: sexual-orientation information is most likely to be revealed, while self-harm information is least likely on average.Helping others produces the most leakage incentive, whereas gaining money produces the least; factors can conflict, such as self-harm information being disclosed when helping others is emphasized.
- Tier 4 results: In Tier 4 meeting-assistant tasks, leakage is higher for summaries than personal action items, and aggregated error rates remain high across all models, including GPT-4.The authors hypothesize that summaries addressed to all attendees make it harder to identify who should be excluded from the secret.
- Mitigation and error analysis: Chain-of-thought prompting is evaluated as a possible mitigation, while manual inspection finds that many failures acknowledge privacy but cannot operationalize it.Another common failure involves incorrectly assuming that a person has access to a secret despite the scenario stating otherwise.
5 CONCLUSION AND DISCUSSION
The paper identifies contextual privacy leaks as an inference-time risk and argues that fundamental, theory-of-mind-informed solutions are needed for real-world deployment.
- Conclusion and Discussion: CONFAIDE evaluates contextual privacy leaks and privacy reasoning across increasingly complex information-flow contexts.The benchmark uses tiers whose seed components define context and increase in complexity, with scenarios involving information types, actors, uses, relationships, and incentives.
- Conclusion and Discussion: Inference-time privacy remains underdefined, with unexplored risks including leakage of in-context examples and conflicts among modalities in multimodal models.The authors emphasize changes to model deployment and use pipelines while noting that they only scratch the surface of possible inference-time concerns.
- Conclusion and Discussion: Ad hoc safeguards such as privacy-inducing prompts, chain-of-thought reasoning, and output filters are insufficient for contextual privacy reasoning.The paper calls for more fundamental and principled inference-time approaches.
- Conclusion and Discussion: The benchmark does not determine universally correct disclosure decisions because privacy judgments can depend on moral and cultural aspects of interactions.The authors instead use contextual integrity to probe privacy reasoning and encourage further study of moral incentives for secret revealing.
- Conclusion and Discussion: People may share personal or corporate confidential information more readily with AI models because they believe their disclosures will remain secure.The paper presents this human-AI interaction pattern as an additional concern motivating trustworthy AI research.
A.5 HUMAN ANNOTATIONS
The human-annotation appendix reports agreement statistics, evaluates the safety and coherence of synthesized scenarios, and documents limits on annotator and worker information.
- Human Annotations: Pair-wise agreement for tiers 1, 2.a, 2.b, and 3 was 70.7%, 76.9%, 74.6%, and 90.8%, respectively.
- Human Annotations: Krippendorff’s alpha for tiers 1, 2.a, 2.b, and 3 was 0.23, 0.19, 0.34, and 0.07, respectively.
- Human Annotations: Out of 270 tier-3 scenarios, only 2 received a majority safety vote and none received a majority coherence vote.The authors plan to drop those scenarios when releasing the dataset.
- Human Annotations: Human expectation and response-preference scores cannot be traced to individual workers because crowdworker IDs were not disclosed.
- Human Annotations: The authors did not collect background information on the Amazon Mechanical Turk annotators.They relate their annotations to a prior study that collected demographic and privacy-concern information.
A.6 SUMMARY OF BENCHMARK STATISTICS
The appendix directs readers to Table 8 for the number of benchmark examples in each tier.
- Summary of Benchmark Statistics: The number of examples in each benchmark tier is summarized in Table 8.
B.1 SUMMARY OF EVALUATION METRICS
The appendix introduces Table 9 as the summary of metrics used to evaluate the models.
- Summary of Evaluation Metrics: Table 9 summarizes the metrics used before the appendix presents detailed result breakdowns.
B.2 TIERS 1-2
The heatmaps show models becoming less conservative as tiers progress, with GPT-4 more conservative than ChatGPT and ChatGPT more conservative than Davinci. Tier 2.a reveals a particularly large human–model disagreement for SSN-sharing contexts.
- B.2 TIERS 1-2: Models become less conservative as tiers progress, while GPT-4 remains more conservative than ChatGPT and ChatGPT more conservative than Davinci.The heatmaps become brighter or more red as tiers progress.
- B.2 TIERS 1-2: Tier 2.a shows the largest human–model judgment gap for SSN-sharing contexts.The highlighted example concerns sharing SSN with insurance for a non-commercial purpose.
B.4 TIER 4
Tier 4 evaluates action-item generation and shows that GPT-4 can reveal private information through nuanced wording that exact string matching misses. Leakage also increases when privacy-inducing instructions are removed.
- B.4 TIER 4: GPT-4 consistently reveals surprise events to people who are not supposed to know them, whereas models leak health issues less frequently.The surprise leakage persisted in generated action items, including wording such as “attend your surprise birthday party.”
- B.4 TIER 4: Figure 5 breaks down Tier 4 privacy expectations across contextual factors for humans and models.The figure presents the Tier 4 contextual-factor breakdown referenced in this appendix.
- B.4 TIER 4: Removing privacy-inducing instructions increases leakage across the Tier 4 results.Figure 6 compares all models with and without privacy prompts.
- B.4 TIER 4: GPT-4 outperforms ChatGPT by a significant margin on ChatGPT-generated Tier 4 scenarios, and GPT-4’s scores improve relative to GPT-4-generated scenarios.The comparison tests whether familiarity with GPT-4-generated scenarios affects evaluation.
B.5 SUMMARY TABLES: WORST/AVERAGE CASE, WITH/WITHOUT PRIVACY PROMPTS
The appendix provides summary tables and heatmaps for Tier 3 and Tier 4 across worst- and average-case settings, with and without privacy-preserving prompts. Across these breakdowns, removing privacy instructions increases leakage, while GPT-4 performs best among the compared models.
- B.5 SUMMARY TABLES: WORST/AVERAGE CASE, WITH/WITHOUT PRIVACY PROMPTS: Leakage is high in average-case Tier 3 results and becomes worse without privacy-preserving instructions.Tables 13 and 14 complement the main-body results and report the corresponding prompt conditions.
- B.5.1 DETAILED HEATMAPS: Removing privacy prompts increases leakage in the Tier 3 heatmaps.Figures 7 and 8 show worst-case results, while Figures 10 and 10 show average-case results.
- B.5.1 DETAILED HEATMAPS: Figure 7 reports worst-case string-matching leakage for GPT-4 and ChatGPT across contextual factors, with and without privacy prompts.Lower values indicate lower leakage.
- B.5.1 DETAILED HEATMAPS: The appendix includes benchmark statistics, tier samples, evaluation-metric summaries, and contextual cases with the largest human–model judgment gaps.These materials are presented in Tables 7, 8, 9, and 10.
- B.5.1 DETAILED HEATMAPS: Table 11 reports Tier 4 metric values without instructing models to preserve privacy, while Table 12 reports results for ChatGPT-generated scenarios.Both tables provide supplementary Tier 4 comparisons.
- B.5.1 DETAILED HEATMAPS: Figure 10 reports average-case string-matching leakage for GPT-4 and ChatGPT across contextual factors, comparing prompted and unprompted results.Lower values indicate lower leakage; the top row omits privacy prompts and the bottom row includes them.
- B.5.1 DETAILED HEATMAPS: Figure 11 breaks down Tier 4 metrics for GPT-4, where Leak measures leakage ratio and ∼Item measures missing action items.Lower is better for both metrics.