Privacy in Context Is Being Overlooked in the Age of General-Purpose LLMs

1. Introduction

• Users increasingly share sensitive information with LLMs, partly because conversations are personalized and open-ended, interaction is better, or users lack privacy literacy.LLMs are embedded in workflows including coding, legal advice, and medical support.Source: §1 · p. 1
• Prompt-privacy solutions increasingly present themselves as universal safeguards, reflecting a one-size-fits-all approach the paper argues is deeply flawed.The paper describes a goal of a user-facing, plug-and-play framework that can be integrated into any chatbot.Source: §1 · p. 1
• No single prompt-privacy framework can achieve all desirable properties because those properties involve non-trivial tensions, and presenting a unified framework may mislead and jeopardize end users.The paper identifies protection guarantees, performance, efficiency, domain adaptability, and user accessibility as the properties at issue.Source: §1 · pp. 1–2
• The paper reviews the literature, identifies five desirable framework properties and tensions among them, examines common approaches, and recommends application-specific deployment, context-specific evaluation, and better privacy literacy.The five properties are protection guarantees, performance, efficiency, domain adaptability, and user accessibility.Source: §1 · pp. 1–2

2. Anatomy of a Prompt Privacy Framework

• The pipeline places a privacy tool or transform before the query reaches the LLM, with privacy interventions occurring in a local trusted environment.Only the protected query is transmitted.Source: §2 · p. 2
• Prompt privacy covers the user’s query to a third-party LLM, including auxiliary material such as documents, images, code, and demonstrations.The assumed adversary may be in the communication channel or at the LLM, with the user query as the leakage target.Source: §2 · p. 2
• The discussion excludes leakage whose privacy target is not the query itself, including training-data extraction, model-inversion attacks, jailbreaking, and backdoor behavior.These phenomena remain out of scope even when they occur during inference.Source: §2 · p. 2
• Figure 3 annotates prompt-privacy literature and publicly available tools to show evidence of drift away from privacy in context.Source: Figure 3 · p. 3

3.1. Prompt Privacy as a Distinct Phenomenon

• General-purpose LLMs serve tasks from grammatical assistance to sensitive medical or legal guidance, making a single privacy framework attractive but liable to overlook context.The paper distinguishes this challenge from earlier application-specific systems, where the deployment context was fixed and developers chose privacy guardrails for that setting.Source: §3.1 · p. 3

3.2. Evidence of the Drift in Academic Literature

• The literature review filtered 228 papers drawn from four recent surveys to 31 prompt-privacy papers, which were read and annotated for privacy target, use case, technique, and discussion of limitations.The paper also frames this systematic study as evidence of the drift toward unified frameworks.Source: §3.2 · p. 3, §3 · p. 3
• More than half of reviewed papers target the entire prompt or its embedding, over three-fourths propose frameworks for all LLM use cases, and more than half omit discussion of limitations or failure cases.The authors characterize these patterns as a drift toward unified frameworks often lacking contextual grounding.Source: §3.2 · pp. 3–4

3.3. Evidence of the Drift in Prompt Privacy Tools

• All 11 publicly available prompt-privacy tools describe general-purpose chatbot use, and none target specialized contexts.Close to half prioritize local LLM deployment; the rest use sanitization or cryptographic deployment in Trusted Execution Environments, while differential privacy appears in literature but not these tools.Source: §3.3 · p. 4

4. Desirable Properties of Prompt Privacy

• The paper’s framework evaluates prompt-privacy approaches through three quantifiable and two qualitative properties.The subsequent distinctions are between protection guarantees, performance, efficiency, domain adaptability, and user accessibility.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 4

4.1. Quantifiable Properties

• Protection guarantees concern limiting information an adversary can extract, with the appropriate protection depending on context, threat model, and attacker knowledge.Different mechanisms may be needed for honest-but-curious providers and malicious attackers.Source: §4.1 · p. 4
• Performance concerns whether a framework preserves the utility of direct LLM access, including output fidelity, actionability, or factual accuracy and consistency.Source: §4.1 · p. 4
• Efficiency concerns the privacy tool’s computational cost, latency, and ability to scale without hindering LLM use.Source: §4.1 · p. 4

4.2. Qualitative Properties

• Domain adaptability means adjusting protection expectations, adversaries, language, or conversational formats across domains and LLM use cases.Source: §4.2 · p. 4
• User accessibility includes transparency and the cognitive burden of fitting a framework into everyday workflows.Understanding query transformations can help users check preserved meaning or debug suboptimal outputs; widely adopted tools often use sanitization, which prioritizes accessibility.Source: §4.2 · pp. 4–5
• The qualitative properties operate above the quantifiable ones: adaptability concerns balancing protection, performance, and efficiency across domains, while accessibility concerns making those relationships understandable and controllable.Source: §4.2 · p. 5

5. User Profiles as Motivating Examples

• The paper uses three motivating examples as recurring reference points for its arguments.Source: §5 · p. 5
• Avery wants a quick marathon-planning draft but worries that leaked or sold chat history could enable targeted ads.Avery is willing to verify the suggestions independently and is not concerned about their accuracy.Source: §5 · p. 5
• Sasha needs a highly accurate work companion with strong privacy protections for confidential documents, running through company infrastructure and a third-party LLM API.The company infrastructure is trusted, but the third-party LLM is not.Source: §5 · p. 5
• Tao wants a writing assistant for casual personal communications that avoids sharing unnecessary identifying information, while prioritizing accuracy and speed over strict guarantees.Tao is a non-native English speaker.Source: §5 · p. 5

6. Drawbacks of Existing Frameworks

• The paper groups prompt-privacy approaches into sanitization, differential privacy, cryptographic methods, and attempts to operationalize contextual integrity, while distinguishing techniques from concepts and theory.It uses “differential privacy” broadly for techniques with formal DP guarantees and “contextual integrity” for attempts to operationalize that social theory.Source: §6 · p. 5, Figure 4 · p. 6

6.1. Sanitization: A Misplaced Sense of Safety

• Sanitization removes, masks, or generalizes prompt content, but lexical methods miss semantic and relational cues and may not generalize across domains.Surface masking can leave residual cues that adversaries use to reconstruct or infer sensitive information; user-facing privacy-utility controls also assume a single continuum despite contextual risks and varying interpretations.Source: §6.1 · pp. 5–6

6.2. Differential Privacy: Promises and Pitfalls

• Differential privacy offers formal protection, but its relative bounds can leave information inferable when it is common across neighboring prompts.Defining sensitive units and meaningful prompt similarity is difficult; conversation-level protection can require prohibitive noise, while word-level removal may leave sensitive context inferable.Source: §6.2 · pp. 6–7

6.3. Cryptography: A Costly Alternative

• Homomorphic encryption and secure multi-party computation protect prompts through encrypted computation or distributed processing, respectively.Source: §6.3 · p. 7
• Homomorphic encryption handles linear operations more efficiently than nonlinear operations, whose processing or approximation costs mean it does not scale well for LLMs.Source: §6.3 · p. 7
• Cryptographic approaches require specialized infrastructure and substantial re-engineering; even hybrid methods retain communication costs that make them too expensive for LLMs.MPC needs multiple non-colluding servers, while existing LLM infrastructure does not support these techniques out of the box.Source: §6.3 · p. 7

6.4. Contextual Integrity: A Distant Ideal

• Contextual integrity assesses whether information flow is appropriate for its context, including who shares what with whom, why, and under which transmission principles.Source: §6.4 · p. 7
• Operationalizing contextual integrity often encodes fixed norms or borrows sanitization, limiting adaptation to changing social contexts and diverse communication.The authors argue that contextual integrity is a social theory requiring more than purely algorithmic solutions.Source: §6.4 · pp. 7–8

7. Underlying Tensions in Prompt Privacy

• The authors identify tensions within prompt privacy that they argue would prevent a unified framework even in the future.They focus on tensions specific to prompt privacy after acknowledging the broader privacy–utility–efficiency trade-off.Source: §7 · p. 8, §7.1 · p. 8

7.2. Natural Language or Embedding Representations

• Language-to-embedding transformations create a disconnect that puts protection guarantees and performance in tension.Semantic similarity in language may not align with similarity in representation space.Source: §7.2 · p. 8
• Representation-based methods can provide formal guarantees only in representation space, requiring over-protection there to protect language-space information.The authors associate this with degraded performance for differential privacy and reduced efficiency for cryptography.Source: §7.2 · p. 8
• Language-space methods such as sanitization can maintain performance but lack formal protection guarantees.The authors characterize this as a choice between stronger representational-space guarantees and better language-space performance or efficiency.Source: §7.2 · p. 8
• The authors describe the choice between these spaces as likely to remain fundamentally irreconcilable, even in the long term.They connect this to the symbol grounding problem, or its modern form, the vector grounding problem [60].Source: §7.2 · p. 8

7.3. User Accessibility or Domain Adaptability

• LLMs’ breadth of use creates varied domain privacy requirements, while adapting across them demands substantial privacy literacy from users.The authors frame accessibility and comprehensive domain adaptability as opposing demands.Source: §7.3 · p. 8
• Accessible sanitization struggles to adapt across domains because terminology, social norms, and definitions of sensitive information vary by context.Contextual integrity is designed for cross-domain adaptability but requires domain-specific expertise and understanding of how privacy situations and sensitivity are defined.Source: §7.3 · pp. 8–9
• The authors argue that frameworks combining accessibility with comprehensive adaptability will face difficult trade-offs as LLMs enter domains with unique privacy requirements.They describe the tension as inherent to prompt privacy, not limited to current approaches, and relate it to the “convention vs configuration” argument.Source: §7.3 · pp. 8–9

• The authors conclude that no current prompt privacy method satisfies all desirable properties, while deeper tensions suggest these limits may persist.They argue that treating LLM privacy as uniform risks real harm by giving users an inaccurate understanding of their chosen techniques’ protections.Source: §8 · p. 9

• A counterargument is that imperfect unified frameworks may be preferable to none, since context-specific choices could overwhelm users.The authors acknowledge that a baseline can be important, but say an imperfect framework is not necessarily preferable to having none.Source: §9 · p. 9
• An imperfect framework may create false security, making users more complacent and more willing to share sensitive information.The authors say framework limitations should be communicated clearly and explicitly.Source: §9 · p. 9
• The authors reject waiting for unified frameworks to evolve naturally, because prompt privacy research norms have already carried into commercial tools marketed as one-stop solutions.They describe intervention as necessary and urgent, while making no claim about how the field might evolve.Source: §9 · p. 9
• A further view is that contextual frameworks cannot stand alone as technical solutions and require broader ecosystem-level sociotechnical efforts [81].Source: §9 · p. 9

10. Where do we go from here?

• The authors recommend application- and context-specific solutions alongside stronger user privacy literacy, rather than relying on a universal framework.They say literacy helps users select appropriate solutions and understand their limitations.Source: §10 · p. 9

10.1. Context-Specific Evaluations

• Choosing a privacy tool starts with identifying an application’s requirements, relevant risks, and acceptable trade-offs.The authors point to contextual integrity as potentially useful in this process.Source: §10.1 · p. 9
• The authors say it remains unclear whether differential privacy can sufficiently protect Avery from targeted ads, or when sanitization becomes problematic in Tao’s email writing.They use these examples to illustrate the need to evaluate frameworks in specific contexts.Source: §10.1 · p. 10

We strongly recommend that the community move toward

• Context-specific evaluation should assess frameworks’ strengths, limitations, and suitability for particular settings.The authors report that less than a quarter of prompt privacy work ties itself to a specific application.Source: We strongly recommend that the… · p. 10, §10.1 · p. 10

10.2. Application-Specific Deployment of LLMs

• Separating applications with substantially different privacy requirements can allow providers to tailor safeguards to sensitive uses such as healthcare.The authors do not endorse healthcare use of LLMs and use it only to illustrate application-specific deployment.Source: §10.2 · p. 10
• Developers should move toward application-specific deployment wherever reasonably possible, avoiding the trap of a unified privacy framework.Source: §10.2 · p. 10

10.3. Fostering User Awareness and Privacy Literacy

• Privacy literacy can help employees recognize confidential information not to share and choose local or third-party LLMs according to context severity.The authors present literacy and accurate mental models of data handling as essential to safer, trustworthy interactions.Source: §10.3 · p. 10

A.1. Academic Literature

• 228 papers from four surveys were deduplicated, then manually filtered to 31 papers focused on protecting inference-time LLM input queries or parts of them.The survey selections included all papers in two surveys and selected attack-stage or in-context-learning categories from the other two.Source: §A.1 · p. 17
• The 31-paper annotations address the privacy target or adversary, targeted use case, discussed limitations or contexts where a framework may not work, and technique category.The annotations are collected in Table 1.Source: §A.1 · p. 17, Table 1 · p. 18
• The coding considered each paper’s central framing, methodological commitments, and conceptual narrative, rather than isolated experiments or section headings alone.A limitations section alone did not count; the review looked for discussion of when a framework might fail.Source: §A.1 · p. 17

A.2. Prompt Privacy Tools

• The tool search began with an online privacy-tools collection and expanded through web searches for “prompt privacy” and “LLM inference privacy.”Searches included Google, Product Hunt, and the Chrome Web Store.Source: §A.2 · p. 17
• The 11 tool annotations record whether each targets a use case, whether it is free or paid, and its broad technique category.The annotations are collected in Table 2.Source: §A.2 · p. 17, Table 2 · p. 19
• Table 1 contains annotations for all 31 reviewed papers.Source: Table 1 · p. 18