PaperlayerView exact revision on arXiv

Source-linked AI summary

Operationalizing Contextual Integrity in Privacy-Conscious Assistants

Sahra Ghalebikesabi, Eugene Bagdasaryan, Ren Yi, Itay Yona, Ilia Shumailov, Aneesh Pappu, Chongyang Shi, Laura Weidinger, Robert Stanforth, Leonard Berrada, Pushmeet Kohli, Po-Sen Huang, Borja Balle

arXiv:2408.02373v2cs.AI

TL;DR

Information-sharing assistants can use personal data to act for users, but ensuring that their autonomous sharing meets privacy expectations is challenging. This paper operationalizes contextual integrity (CI) in assistants and evaluates CI-based reasoning using a human-annotated form-filling benchmark. The evaluation reports that CI-based assistants achieve the best performance among the options investigated.

  • Problem

    Assistants that autonomously share users’ personal information with third parties need to meet context-dependent privacy expectations, but social norms are difficult to capture and LLMs lack a framework for CI reasoning.

    Source: §1 · p. 1

  • Method

    The paper operationalizes CI by eliciting norms through a user study and asking assistants to infer Information Flow Cards and reason about whether sharing is appropriate.

    Source: §1 · pp. 2–3, §4.4 · p. 8

  • Results

    CI-based assistants achieve the best performance among the investigated options, with evaluation indicating reduced privacy leakage without major deterioration in utility.

    Source: §1 · p. 3, §6 · p. 13

  • Takeaways & Limitations

    Human annotations of synthetic forms offer a path toward larger benchmarks for evaluating CI in assistants.

    Source: §1 · p. 2, §6.1 · p. 13

    The evaluation focuses on form filling with synthetic forms and user data, while broader application domains and information norms remain future work.

    Source: §1 · p. 2, §6.1 · p. 13

Abstract

from arXiv · show

Advanced AI assistants combine frontier LLMs and tool access to autonomously perform complex tasks on behalf of users. While the helpfulness of such assistants can increase dramatically with access to user information including emails and documents, this raises privacy concerns about assistants sharing inappropriate information with third parties without user supervision. To steer information-sharing assistants to behave in accordance with privacy expectations, we propose to operationalize contextual integrity (CI), a framework that equates privacy with the appropriate flow of information in a given context. In particular, we design and evaluate a number of strategies to steer assistants' information-sharing actions to be CI compliant. Our evaluation is based on a novel form filling benchmark composed of human annotations of common webform applications, and it reveals that prompting frontier LLMs to perform CI-based reasoning yields strong results.

1. Introduction

Autonomous assistants can help users across many tasks, but access to personal information and third-party interactions make privacy expectations difficult to maintain. This work uses contextual integrity (CI) to collect norms and guide sharing decisions, then evaluates the approach in form filling.

1. Introduction

  • The study asks how to collect contextual privacy norms for evaluating assistants and how to design assistants that conform to those norms.The paper identifies social norms as difficult to capture and CI reasoning as lacking a framework for LLMs.Source: §1 · p. 2
  • The paper operationalizes CI by having assistants infer an Information Flow Card and reason about whether a potential information flow is appropriate.The assistant should share only when the action is appropriate under applicable norms.Source: §1 · pp. 2–3
  • Form filling provides a tractable assistive task and a proxy for API calling; synthetic forms and user data are paired with a real user study to elicit social norms.Its structured outputs enable more precise quantitative analysis than unstructured text generation.Source: §1 · p. 2
  • CI-based reasoning achieves better privacy and utility than the alternatives evaluated on a benchmark combining synthetic forms with human annotations.The benchmark focuses on form-filling assistants.Source: §1 · p. 3
  • The work differs from prior CI-related evaluations by studying assistive rather than conversational tasks and using human annotations rather than model-produced benchmark labels.The cited comparison concerns Mireshghallah et al. [25] and Bagdasaryan et al. [5].Source: §1 · p. 3

2. Design and Evaluation of Information-Sharing Assistants

The paper formalizes information-sharing assistants as systems that use user information to produce outputs for third parties, and defines utility and privacy leakage against task-specific ground truth. Its evaluation model simplifies interactions and assumes structured user information and single-key form fields.

2. Design and Evaluation of Information-Sharing Assistants

  • The framework covers tasks such as form filling, API calls, and email replies, modeling assistants that consume user information and share outputs with third parties.The assistant produces k strings that contain information from its available user data.Source: §2 · p. 3
  • For simplicity, the model considers a single information-sharing action and represents the sent message as a task-specific template filled with assistant outputs.The task may require multiple interactions in practice, but the formal model limits itself to one sharing action.Source: §2 · p. 4
  • Privacy leakage is information shared beyond what is necessary for the task under user expectations, including information requested by a third party that conflicts with those expectations.Oversharing can result from misjudging necessity or complying with an incompatible third-party request.Source: §2 · p. 4
  • The model assumes all task-relevant information is available in a fixed-key list of user data, and each template blank depends on one key.Blanks requiring multiple values can be decomposed into selection followed by post-processing.Source: §2 · p. 4
  • Utility measures correctly generated outputs where an answer is expected, while leakage measures outputs containing information where ground truth requires silence.The leakage metric counts information sharing even when the output is incorrect, making it stringent.Source: §2 · p. 5

3. Designing Privacy-Conscious Information Sharing Assistants

The section proposes assistant designs that decide whether information should flow in a given context, using contextual integrity (CI) as a basis for judging appropriateness. Designs progress from self-censoring to supervised decisions, including a CI-based supervisor that reasons with an information flow card.

3. Designing Privacy-Conscious Information Sharing Assistants

  • CI judges an information flow by its attributes, context, actor relationships, and purpose; a flow is appropriate when no norm forbids it and at least one allows it.The norms represent widely accepted societal expectations, rather than individual preferences.Source: §3 · p. 5

3.1. Assistant Designs

  • Self-censoring lets the assistant leave a form field blank when it judges sharing inappropriate in that context.For a prompted LLM, the prompt can ask it to refuse inappropriate information.Source: §3.1 · pp. 5–6
  • A binary supervisor decides whether each field should be filled and controls which fields the filler can access.This separates privacy decisions from the filler’s information-sharing task.Source: §3.1 · p. 6
  • A reasoning supervisor explains whether a field should be filled, then extracts a binary decision from that explanation.The explanation precedes the fill-or-not-fill judgement.Source: §3.1 · p. 6
  • A CI-based reasoning supervisor grounds its decision in applicable norms and an information flow card (IFC) capturing relevant flow features.The IFC supplies features CI considers necessary for judging appropriateness.Source: §3.1 · p. 6

4. Evaluation Methodology for Form-Filling Assistants

The evaluation uses synthetic form-filling tasks and personas, with human ratings of whether information fields are necessary or relevant in each form context. The benchmark also incorporates identified legal norms and examines disagreement among raters.

4. Evaluation Methodology for Form-Filling Assistants

  • The benchmark pairs online forms with synthetic personal information and human judgements of whether fields are appropriate to fill in context.Synthetic personas address the difficulty of evaluating on real user data, which forms often request but public datasets do not provide.Source: §4 · p. 6

4.1. Implementation of Form-Filling Assistants

  • Self-censoring uses one prompted LLM, while supervised assistants query a supervisor and filler separately, with an extraction module turning the supervisor’s response into a binary trigger.Reasoning supervisors request explanations; CI-based supervisors ask for explanations framed by the information flow card.Source: §4.1 · p. 7

4.2. Synthetic Persona Generation

  • Personas assign values to 51 manually defined information keys, generated from one of 18 high-level descriptions using Gemini Ultra.Authors reviewed personas for consistency and missing values, and restricted them to US residents to limit ambiguity in applicable norms.Source: §4.2 · p. 7

4.3. Synthetic Form Generation

  • Fourteen form scenarios were expanded into 126 forms by generating descriptions and titles with three seeds per scenario and two paraphrases per title-description pair.The scenarios include work-event registration, opening a checking account, and newsletter subscription.Source: §4.3 · p. 7
  • Fields draw on persona keys; five phrasings per key were reviewed, yielding 193 phrasings, and each form samples seven relevant keys with randomized wording.The key-to-field mapping is withheld from the assistant and used for evaluation; scenario relevance comes from human annotations.Source: §4.3 · pp. 7–8

4.4. Human Annotations

  • Eight raters labelled all 714 key-scenario pairs on a five-point scale, defining “necessary” as required to achieve the form’s purpose and “relevant” as occurring in some forms but possibly optional.The figure plots each rater’s ratio of “necessary” to “relevant” labels across applications F1–F14.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Because raters can disagree even when asked to judge population-wide norms, a key was labelled necessary only when no rater disagreed; this yielded 50 Yes, 460 No, and 232 Unsure labels.The figure shows the necessary-to-relevant label ratio by application and rater.Source: §4.4 · pp. 8–9, Figure 4 · p. 8
  • For identified regulated cases, legal-norm labels replace annotator labels, changing the necessary-label distribution to 87 Yes, 470 No, and 185 Unsure.Examples include requirements associated with FERPA, the IRC, and HIPAA.Source: §4.4 · p. 9

5. Quantitative Analysis

The evaluation compares form-filling assistants across models, privacy leakage, and utility. Supervisors improve on self-censoring, while reasoning and CI-based supervision show different strengths; reasoning-based designs also exhibit robustness patterns across forms and model sizes.

Privacy and utility evaluation

  • With Gemini Ultra, the CI-based reasoning supervisor outperforms other assistants on both utility and privacy by a small margin, while reasoning supervision reduces leakage and can increase utility.The figure reports utility and privacy leakage for five assistant types; the prose notes that reasoning supervision is best for privacy leakage and CI-based supervision for utility.Source: §5 · p. 9, Figure 6 · p. 10
  • Larger models perform better than smaller ones except for Gemma, and further reasoning supervision consistently decreases privacy leakage.Across Gemini Pro, Gemma, and Mistral, the qualitative behavior is similar, although overall performance is slightly worse than with Gemini Ultra.Source: §5 · p. 9, Table 1 · p. 11

Robustness analysis

  • CI-based supervisor decisions vary less across paraphrasings, while reasoning-based supervisors produce more consistent outputs across Gemini Ultra and Pro.Agreement is overall higher between reasoning-based assistant instantiations.Source: §5 · p. 11, Figure 7 · p. 12, Figure 9 · p. 13

Raters disagreement and model uncertainty

  • Gemini Ultra uncertainty is positively correlated with rater disagreement for all prompting approaches except the CI approach; this pattern is not observed for Gemini Pro.Uncertainty is measured by the log likelihood of the model response being “Unsure.”Source: §5 · p. 11

Additional experiments and ablations

  • Appendix experiments examine prompt choices, persona inclusion, performance by information-key category, annotator disagreement, and personalization toward individual raters.The authors also explore effects of few-shot examples and privacy-requirement phrasing.Source: §5 · p. 13

6. Conclusion

The paper presents contextual integrity (CI) as a basis for aligning information-sharing assistants with privacy expectations and reports that CI-based assistants perform best among those investigated. It identifies limits in the synthetic-data evaluation, prompting-based designs, and the complexity of applying CI across contexts.

6. Conclusion

  • The evaluation is a first step toward showing existing models can reduce privacy leakage without major utility deterioration, with CI-based assistants performing best among the options investigated.The conclusion also describes the work as introducing use cases and synthetic personas to add real-world complexity to evaluation.Source: §6 · p. 13

6.1. Limitations and Future Work

  • Synthetic data and the limited human interaction evaluation require further experiments with humans; future work also includes studying users’ perceptions of CI-based form filling.Future benchmark work should support generalization across contexts, uncertainty handling, and CI beyond form filling.Source: §6.1 · p. 13
  • The evaluation covers prompt engineering and few-shot prompting, while fine-tuning, protection against malicious third parties, and user-interface design remain future research directions.The paper notes that prompt-based techniques suit its low-data regime.Source: §6.1 · p. 14
  • CI’s relevant information-flow features can be difficult to enumerate across an assistant’s application range, and personalization or cultural adaptation may require going beyond CI.The paper contrasts this challenge with narrower applications where feature taxonomies can be manually constructed.Source: §6.1 · p. 14

Appendix: Overview

  • Appendix A and Appendix B provide additional experimental details and results, while Appendix C describes benchmark construction.The submission checklist is in the supplementary material.Source: Appendix: Overview · p. 18

A. Experimental Details

The experimental prompts combine form details and requested data keys with user information when available. Few-shot examples are randomly sampled from a fixed set, and the appendix shows examples for different assistant types and form fields.

A. Experimental Details

  • Base prompts append available user information and the form to a prompt containing the data key, form heading, and form description.The prompt structure uses DATA_KEY, FORM_HEADING, and FORM_DESCRIPTION placeholders.Source: §A.1 · p. 19
  • User information is represented as key-value pairs rendered as “{DATA_KEY}: {DATA_VALUE}” entries.The passage gives the format for N key-value pairs.Source: §A.1 · p. 19
  • Few-shot examples are randomly sampled from a fixed list of five, with examples spanning rental, travel visa, mortgage, complaint, and competition forms.The shown prompts include varied user details and answers such as “Unsure,” “no,” and “yes.”Source: §A.1 · pp. 19–20

A.2. Compute

The appendix reports estimated model-call counts for final results and experimentation phases. It also points to tables covering prompt templates and privacy-leakage and utility measures.

A.2. Compute

  • 381,024 Gemini Pro calls and 111,345 Gemini Ultra calls were estimated for the final set of results.Source: §A.2 · p. 23
  • 111,132 Gemini Pro calls and 286,407 Gemini Ultra calls were estimated for experimentation phases.Source: §A.2 · p. 23
  • Table 4 presents base prompts for different assistant types, while Table 6 reports privacy leakage and utility measures for a subset of best-performing assistants across data-key categories.Table 6 refers readers to Table 11 for category descriptions.Source: Table 4 · p. 24, Table 6 · p. 25

B. Additional Experimental Results

Additional analyses examine performance categories, model uncertainty in relation to rater disagreement, and variations in user information and prompting. A further table addresses sensitive keys queried in forms rated irrelevant for them.

B. Additional Experimental Results

B.1. Adjusting to User Preferences

This appendix tests adjusting CI-based assistants to an individual rater’s privacy preferences; a restrictive preference prompt lowers privacy leakage at the cost of utility.

B.1. Adjusting to User Preferences

  • The adaptation targets Rater 1, whose labels included the lowest overall ratio of necessary judgments.The authors use this rater’s preferences as a basis for adapting the assistant.Source: §B.1 · p. 25
  • The preference prompt represents a person outside the US who prefers sharing less data when filling forms.The prompt instructs a CI-based supervisor to follow those preferences.Source: §B.1 · p. 27
  • The restrictive prompt lowered privacy leakage for Rater 1, with reduced utility as the trade-off.Table 9 reports privacy leakage and utility when assistants are prompted according to Rater 1’s preferences.Source: §B.1 · p. 29, Table 9 · p. 30

C. Benchmark Construction Details

The benchmark construction generates persona information and varied webform applications, then paraphrases their titles and descriptions.

C.1. Persona Generation

  • Eighteen high-level personas were manually generated to guide the persona-generation process.The listed personas span roles, demographic groups, and circumstances.Source: §C.1 · p. 31
  • Gemini Ultra generated alternative names for data fields, designed to make each field seem necessary without creating privacy problems.The prompt allowed spelling mistakes and word perturbations.Source: §C.1 · p. 31
  • Gemini Ultra filled data-tag values; annotators reviewed them, and all persona values were assumed to be US-based.Null values were filtered, and human annotators reviewed generated personas for consistency and null values.Source: §C.1 · p. 36

C.2. Form Generation

  • The benchmark uses 14 manually curated form-application subjects, spanning financial, employment, medical, subscription, and event forms.Gemini Ultra generated descriptions for each subject and titles for those descriptions.Source: §C.2 · p. 37
  • Form titles and descriptions were paraphrased with instructions to preserve meaning while varying wording, names, order, capitalization, and errors.The paraphrasing examples include changed proprietary names and wording.Source: §C.2 · pp. 37–38

C.3. Categorisation of Data Keys

The appendix categorizes data keys using four schemes and describes how human ratings and agreement patterns were used to construct labels.

C.3. Categorisation of Data Keys

  • Data keys are classified by sensitivity, information type, need-to-know confidentiality, and identifiability.Identifiability distinguishes directly identifiable, indirectly identifiable, and non-identifiable fields.Source: §C.3 · p. 39, Table 11 · p. 40

C.4. Human Annotations

  • Raters labeled each application–data-key combination for necessity and relevance on a five-point scale; the study collected eight rating sets per combination.The paper considered several aggregation mappings and used “no veto (normed)” throughout.Source: §C.4 · p. 39
  • Agreement varied by field: raters tended to agree on pregnancy and sexual orientation, but differed on number of children and credit history.The figure attributes some credit-history disagreement to cultural differences in income sensitivity.Source: Figure 16 · p. 44
Loading 2408.02373v2…