PaperlayerView exact revision on arXiv

Source-linked AI summary

PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, Diyi Yang

arXiv:2409.00138v3cs.CLcs.AIcs.CR

TL;DR

LM privacy evaluation often relies on probing questions, although realistic agent actions can expose privacy-norm violations during communication. PrivacyLens constructs contextual seeds, expressive vignettes, and agent trajectories for multi-level evaluation, revealing frequent leakage despite strong probing performance and privacy-oriented prompting. The framework can also expand seeds into diverse trajectories for red-teaming, but its evidence is limited to U.S. norms and a GPT-4-generated trajectory setup focused on LM-mediated communication.

  • Problem

    Existing privacy evaluations may not capture the gap between LM answers to probing questions and behavior in realistic, tool-using communication scenarios.

  • Method

    PrivacyLens extends contextual privacy-sensitive seeds into expressive vignettes and agent trajectories, evaluating both probing judgments and final actions.

  • Results

    GPT-4 leaked information in 25.68% of action-based cases despite privacy-enhancing prompt engineering, while strong LMs could answer nearly all corresponding trajectory-level probing questions correctly.

  • Takeaways & Limitations

    PrivacyLens shows that strong probing performance does not ensure privacy-preserving behavior when LMs execute user instructions in agent setups.

  • Takeaways & Limitations

    The study covers general privacy norms in the United States, uses GPT-4 to collect trajectories while other LMs perform final actions, and focuses on LM-mediated communication.

Abstract

from arXiv · show

As language models (LMs) are widely utilized in personalized communication scenarios (e.g., sending emails, writing social media posts) and endowed with a certain level of agency, ensuring they act in accordance with the contextual privacy norms becomes increasingly critical. However, quantifying the privacy norm awareness of LMs and the emerging privacy risk in LM-mediated communication is challenging due to (1) the contextual and long-tailed nature of privacy-sensitive cases, and (2) the lack of evaluation approaches that capture realistic application scenarios. To address these challenges, we propose PrivacyLens, a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories, enabling multi-level evaluation of privacy leakage in LM agents' actions. We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM performance in answering probing questions and their actual behavior when executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions. We also demonstrate the dynamic nature of PrivacyLens by extending each seed into multiple trajectories to red-team LM privacy leakage risk. Dataset and code are available at https://github.com/SALT-NLP/PrivacyLens.

1 Introduction

PrivacyLens targets unintentional privacy leakage by LM agents in contextual communication, where privacy norms and realistic action-based behavior are difficult to evaluate. It constructs multi-level evaluations and finds that strong probing performance does not prevent leakage during agent actions.

  • Motivation: LM agents can access sensitive data through tools and unintentionally violate contextual privacy norms when communicating on users’ behalf.The risk differs from traditional privacy models involving intentional attackers.
  • Motivation: Probing questions reveal privacy reasoning capabilities, but may not reflect LM behavior in realistic applications.The paper therefore focuses on agents interacting with tools such as calendars and email.
  • Approach: PrivacyLens uses privacy norms to evaluate LM privacy norm awareness in action through a procedural, multi-level framework.Its norms are informed by Contextual Integrity, including the people involved and the transmission method.
  • Approach: 493 seeds were extended into 493 vignettes and 493 trajectories, with 1,479 probing questions used alongside agent evaluation.The seeds came from U.S. regulations, privacy literature, and crowdsourcing.
  • Findings: GPT-4 leaked information in 25.68% of action-based cases despite privacy-enhancing prompt engineering.PrivacyLens also supports generating additional vignettes and trajectories for red-teaming.

2 Related Work

Prior LM privacy research emphasizes memorization and malicious extraction, while agent benchmarks emphasize task performance. PrivacyLens adds unintentional leakage and action-based evaluation to this research landscape.

  • Language Model Privacy: Earlier LM privacy studies mainly examine training-data memorization and extraction by malicious attackers.These risks do not cover all privacy problems arising during inference.
  • Language Model Privacy: LM agents can expose private information in generated text by violating social norms specific to a context.This extends privacy concerns beyond memorization.
  • Evaluating Language Model Agents: Existing agent benchmarks evaluate capabilities across domains such as web environments, games, coding, and social interactions.Task completion alone does not capture consequences of agent actions on users’ behalf.
  • Comparison: PrivacyLens compares prior privacy evaluations with unintentional-leakage and action-based evaluation dimensions.The comparison distinguishes leaks without malicious attackers from actions performed by LM agents.
  • Language Model Assisted Evaluation: PrivacyLens contributes a procedural data-construction framework that uses language models to construct vignettes and agent trajectories for evaluation.It draws on prior language-model-assisted evaluation and red-teaming approaches.

3 PrivacyLens

PrivacyLens converts contextual privacy seeds into detailed vignettes and executable agent trajectories, then evaluates both privacy judgments and final actions. Its risk model treats leakage as norm-violating information gathered during tool use and shared in the final action.

  • 3.1 Risk Model behind PrivacyLens: PrivacyLens models a sender, recipient, and LM agent executing user instructions through tool-use actions and observations.The model excludes malicious actors and focuses on unintentional leakage.
  • 3.2 Data Construction Pipeline: Each privacy-sensitive seed specifies data type, data subject, sender, recipient, and transmission principle.Changing one element can change the expected privacy norm.
  • 3.2 Data Construction Pipeline: Seeds are expanded into detailed vignettes, then used with sandbox simulation to produce executable agent trajectories.Template-based generation adds real-world complexity, while the sandbox supports tools such as calendars and email.
  • 3.2 Data Construction Pipeline: The Surgery Kit refines generated outputs using unit tests and language-model repair instructions.It is used to improve vignette and trajectory quality and remove restricted sensitivity keywords from vignettes.
  • 3.3 Multi-Level Evaluation: PrivacyLens evaluates privacy awareness through both probing questions and agent actions.The probing format asks whether a contextual transmission is acceptable, while action evaluation examines the agent’s final action.
  • 3.3 Multi-Level Evaluation: The action-based leakage rate is the percentage of final actions from which at least one sensitive trajectory item can be inferred.Sensitive items are extracted from each trajectory and judged with an LM-based few-shot classifier.

4 Instantiating PrivacyLens

The PrivacyLens dataset instantiates contextual privacy evaluation with seeds collected from U.S. legal, research, and crowdsourced sources. The framework is extensible to other seed collections and scenarios.

  • Seed Collection: PrivacyLens collects U.S. privacy norms spanning legal, social, specialized-community, and general-public sources.Legal seeds were drawn from 15 U.S. privacy regulations and professional rules.
  • Seed Collection: Crowdsourcing expands seeds by combining online communication activities with social relationships, occupations, data types, and data subjects.Participants generated combinations intended to violate privacy norms.
  • Extensibility: PrivacyLens can convert seeds from other privacy datasets and cultural knowledge bases into contextualized evaluation points.Experiments on these sources further exposed LM privacy leakage across scenarios.

5 Experiments

PrivacyLens evaluates privacy norm awareness through both probing questions and LM-agent actions, measuring leakage alongside helpfulness. Experiments reveal substantial discrepancies between models’ privacy judgments and their executed actions, while dynamic trajectory expansion supports broader red-teaming.

  • Evaluation Setup: The evaluation tests four closed-source and five open-source language models at temperature 0 using probing questions and ReAct-based agent actions.Agents use Basic and Privacy-Enhancing prompts, while action evaluation quantifies leakage and helpfulness.
  • Discrepancy between probing accuracy and action-based evaluation: 27.23% and 38.83% are GPT-4 and Claude-3-Sonnet leakage rates, respectively, despite near-perfect trajectory-level probing accuracy.The same trajectories expose a large discrepancy between judging data transmission and acting on that judgment.
  • Safety-helpfulness trade-off: Llama-3-8B achieves the lowest leakage rate but scores low on helpfulness, showing that safer actions can still fail to execute user instructions effectively.GPT-4 examples likewise include both helpful leakage and privacy-preserving but unhelpful actions.
  • Effect of scaling: Larger models can improve probing performance without improving action-based privacy behavior, and still tend to leak sensitive information in final actions.This pattern appears when comparing Claude-3-Haiku with Claude-3-Sonnet and Llama-3-8B-Instruct with Llama-3-70B-Instruct.
  • Effect of prompt engineering: Privacy-Enhancing Prompts improve probing results but do not significantly improve action-based evaluation, even when stronger models mention privacy in their thoughts.Models may leak information despite explicitly considering privacy before acting.
  • Dynamic Nature of PrivacyLens: PrivacyLens expands seeds into diverse vignettes and trajectories, enabling dynamic evaluation and broader red-teaming of unintentional privacy leakage.A proof of concept converts 10 seeds under five conditions into 50 trajectories, whose results remain useful for action-based assessment.

6 Discussion

PrivacyLens provides an extensible way to evaluate privacy norm awareness in LM actions, showing that strong probing performance does not prevent privacy leakage during agent execution. The framework also supports broader red-teaming, while its current scope and trajectory construction impose important limitations.

  • PrivacyLens quantifies unintentional privacy leakage in LM actions through an extensible evaluation framework.
  • State-of-the-art LMs can perform well on probing questions yet still leak information when executing user instructions in an agent setup.
  • The dataset covers general privacy norms in the United States, while privacy concerns can differ across individuals and cultures.
  • The evaluation focuses on LM-mediated communication and may not generalize to other scenarios, such as web agents interacting with websites.

Checklist

The checklist documents the paper’s contributions, scope, data practices, reproducibility materials, and crowdsourcing procedures. It also records that the dataset contains no personally identifiable or offensive content and that the work focuses on privacy norm awareness in LMs.

  • Contributions and scope: The paper proposes PrivacyLens, a privacy-norm data construction pipeline and evaluation framework for language models.
  • Contributions and scope: The work focuses on evaluating privacy norm awareness rather than theoretical results.
  • Reproducibility: The authors provide code, data, and reproduction instructions through the PrivacyLens repository.
  • Assets and data: The dataset is curated from scratch and does not contain personally identifiable information or offensive content.
  • Data construction: Privacy norms are collected from U.S. regulations, literature, and crowdsourcing, then converted into scenarios using information, relationships, context, and privacy-sensitive reason.
  • Data construction: The scenario-generation format specifies sensitive information, data subject, sender, recipient, context, and privacy-sensitive reason.

B.3 Validation Phase

The validation phase addresses potential noise in seeds extracted by GPT-4 or collected through crowdsourcing by using multiple human annotations and filtering unclear cases.

  • Seeds extracted by GPT-4 or collected through crowdsourcing may be noisy, so the authors conduct a validation phase.
  • Each seed receives three annotations from a team of four authors and one volunteer student.
  • Seeds flagged as unclearly described by any annotator are removed before retaining the validated set.

C More Details of Agent Trajectory Construction

PrivacyLens constructs executable agent trajectories in a sandbox where an LM agent uses communication and personal-data tools against emulator-generated observations. The evaluation removes the final action so different LMs can be compared on the same preceding trajectory, but this design limits validity for non-GPT-4 models and excludes earlier leakage.

  • The sandbox uses GPT-4 to emulate tool execution and generate observations from each seed and vignette.
  • A privacy-enhancing GPT-4 agent interacts with the sandbox to collect trajectories consisting of alternating actions and observations.
  • Available sandbox tools include calendars, email, social media, and personal notebooks for constructing agent trajectories.
  • The final action is removed from each trajectory, allowing different LMs to be evaluated on their final action given the same preceding interactions.
  • This setup may affect the validity of evaluating LMs other than GPT-4 and does not assess privacy leakage occurring before the final action.

D Extensibility of PrivacyLens

PrivacyLens is extensible beyond its main dataset: it can reuse external seed collections and evaluate privacy leakage in action, while supporting culturally varied norms. Results on a repurposed ConfAIde subset show measurable leakage by GPT-4 and Claude-3-Sonnet.

  • PrivacyLens can be instantiated with seed collections beyond its main evaluation dataset.
  • 9.38% of GPT-4 and 21.88% of Claude-3-Sonnet agent cases leak information on the repurposed ConfAIde subset.
  • PrivacyLens can incorporate cultural privacy norms by specifying the data sender’s cultural group, but the CultureBank subset contains only 24 meaningful interpersonal privacy norms.The main dataset’s U.S. focus is a limitation because privacy norms are culturally dependent.

E More Qualitative Examples

The qualitative examples illustrate that privacy leakage can arise from ordinary agent actions, including social-media posting and messaging, and that low aggregate leakage rates may not capture concerning behaviors. The examples also show agents encountering highly sensitive records before deciding how to act.

  • PrivacyLens reports adjusted leakage conditioned on helpful actions, but some concerning behaviors, such as suggesting direct data access or deferring transmission, remain inadequately captured.The paper notes these behaviors even when a model has the lowest reported leakage rate.
  • Llama-3-8B-Instruct posted high-profile embezzlement-defense information after retrieving it from the user’s Notion.
  • The address example involves a domestic-violence survivor whose alternate mailing address is protected through an Address Confidentiality Program.
  • The Slack trajectory exposes a request for Michael Thompson’s address, while the agent’s final message says it found the information and will send it separately.

F Full Prompts

The full prompts operationalize PrivacyLens as a pipeline that expands structured seeds into vignettes, generates underspecified agent instructions, and simulates tool interactions for privacy stress testing. Prompt variations create multiple trajectories for broader red-teaming.

  • F.1.1 Seed to Vignette: PrivacyLens converts a seed tuple into a five-sentence vignette describing the sender, recipient, sharing rationale, sensitive data, and associated entities.The prompt requires sensitive information to appear in context without explicitly labeling it as insensitive.
  • F.1.1 Seed to Vignette: Vignettes are instructed to describe a possible privacy-sensitive context without stating that the transfer has happened or revealing the sender’s decision or plan.
  • F.1.1 Seed to Vignette: The prompt illustrates vignette construction with medical-data and employment-search seeds that situate sensitive information in ordinary workplace contexts.
  • F.1.1 Seed to Vignette: Five contextual conditions vary reciprocal disclosure, legitimate need, relationship closeness, sender excitement, and perceived benefits of sharing.The paper reports that expanding seeds into multiple trajectories supports more holistic red-teaming of unintentional leakage.
  • F.1.2 Generate User Instruction: Adversarial user instructions are designed to be realistic, underspecified about the information to share, and executable by an agent rather than answered with a draft.
  • F.1.3 Sandbox Emulator: The sandbox emulator simulates realistic tool outputs for an agent operating over a detailed plot, while explicitly avoiding real-world effects.Its roles include the user, agent, and simulator, with tools such as calendars, email, social media, and personal notebooks represented in the environment.
  • F.1.3 Sandbox Emulator: The sandbox prompt prioritizes realistic outputs, including personal contact information, and defines the simulator as responsible for tool execution rather than helpful user assistance.

G PrivacyLens Datasheet

The dataset was created to evaluate whether language models recognize privacy norms during action and to quantify unintentional privacy leakage in agentic applications. The supplied datasheet identifies the authors’ SALT lab affiliation and reports no funding.

  • The dataset evaluates LM privacy norm awareness in action and quantifies unintentional privacy leakage risk in agentic applications.
  • The dataset was created by the paper’s authors as part of the Stanford Social and Language Technologies lab.
  • No funding source is reported for the dataset’s creation.

G.2 Composition

PrivacyLens is a self-contained evaluation dataset of privacy-sensitive seeds, contextual vignettes, and executable LM agent trajectories, designed for evaluation rather than training. Its instances are curated from U.S. privacy sources and crowdsourcing, manually reviewed, and labeled with probing questions about privacy-appropriate transmission.

  • Dataset composition: 493 privacy-sensitive seeds, 493 vignettes, and 493 LM agent trajectories comprise the dataset.
  • Data acquisition: Seeds were curated from U.S. privacy regulations, privacy literature on marginalized groups, and crowdsourcing, then expanded into vignettes and trajectories through PrivacyLens.GPT-4 extracted information from regulatory documents and literature, while Prolific participants contributed ideas for seed fields.
  • Dataset composition: Each instance contains a privacy-sensitive seed, a corresponding vignette, and an executable LM agent trajectory.Seeds use five fields: data type, data subject, data sender, data recipient, and transmission principle.
  • Evaluation labels: Each instance has three probing questions evaluating transmission appropriateness at the seed, vignette, and trajectory levels, with “No” as the expected answer.
  • Dataset composition: The dataset is intended solely for evaluation, not training, and is self-contained.
  • Quality and scope: The authors manually reviewed the data points and report no known errors, redundancies, or noise.
Loading 2409.00138v3…