PaperlayerView exact revision on arXiv

Source-linked AI summary

CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data

Zhao Cheng, Diane Wan, Matthew Abueg, Sahra Ghalebikesabi, Ren Yi, Eugene Bagdasarian, Borja Balle, Stefan Mellem, Shawn O'Banion

arXiv:2409.13903v1cs.AI

TL;DR

Inference-time privacy evaluation has often been limited in scope, despite AI assistants’ growing access to user information. CI-Bench provides a synthetic benchmark for assessing contextual information sharing, and its initial evaluations find that models struggle with nuanced scenarios while explicit norms improve appropriateness judgments.

  • Problem

    Inference-time privacy leakage has been little studied, and existing evaluations often focus on narrow applications rather than broader settings.

    Source: Related Work · p. 1

  • Method

    CI-Bench uses the Contextual Integrity framework and a scalable pipeline to create structured scenarios and synthetic conversational data for evaluating information-flow judgments.

    Source: Contributions · p. 2

  • Results

    State-of-the-art models show promising zero-shot performance but struggle with nuanced multi-topic and context-switching scenarios, while explicit context-specific norms improve appropriateness judgments.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    CI-Bench offers a first step toward evaluating prototype assistants and understanding capability gaps in assistant-driven information sharing.

    Source: Discussion · p. 6

    Samples assume stateless, self-contained interactions, and each session contains only one context rather than multiple contexts or transitions.

    Source: Limitations and Future Work · p. 7

Abstract

from arXiv · show

Advances in generative AI point towards a new era of personalized applications that perform diverse tasks on behalf of users. While general AI assistants have yet to fully emerge, their potential to share personal data raises significant privacy challenges. This paper introduces CI-Bench, a comprehensive synthetic benchmark for evaluating the ability of AI assistants to protect personal information during model inference. Leveraging the Contextual Integrity framework, our benchmark enables systematic assessment of information flow across important context dimensions, including roles, information types, and transmission principles. We present a novel, scalable, multi-step synthetic data pipeline for generating natural communications, including dialogues and emails. Unlike previous work with smaller, narrowly focused evaluations, we present a novel, scalable, multi-step data pipeline that synthetically generates natural communications, including dialogues and emails, which we use to generate 44 thousand test samples across eight domains. Additionally, we formulate and evaluate a naive AI assistant to demonstrate the need for further study and careful training towards personal assistant tasks. We envision CI-Bench as a valuable tool for guiding future language model development, deployment, system design, and dataset construction, ultimately contributing to the development of AI assistants that align with users' privacy expectations.

Introduction

AI assistants can mediate information sharing using user data, making privacy depend on whether information flows fit the norms of the interaction. CI-Bench addresses limited, narrowly scoped inference-time privacy evaluations with a broad benchmark and synthetic data pipeline.

Background

  • Contextual Integrity evaluates whether information flows are appropriate under context-specific norms, considering actors, information types, and transmission principles.The paper also describes context features and individual interaction history as relevant to interpreting information flows.Source: Background · p. 1

Related Work

  • Prior inference-time privacy evaluations often cover only one domain, data format, or platform, limiting evidence about generalization to broader applications.Other benchmarks may ignore context-specific norms or lack systematic distinctions between identifying sensitive data and judging appropriate information flow.Source: Related Work · pp. 1–2

Contributions

  • CI-Bench assesses information-flow appropriateness across structured scenarios and unstructured dialogues or emails, with tasks for context understanding, norm identification, and judgment.Its scalable pipeline uses real-world structured data to generate synthetic conversational data.Source: Contributions · p. 2
  • Prototype evaluations found room for improvement in appropriateness judgments, difficulty for small models in context understanding, and better accuracy with well-defined data-flow rules.The benchmark is proposed for evaluation, future model and system development, and dataset construction for post-training and fine-tuning.Source: Contributions · p. 2

Benchmark

The benchmark decomposes information-flow mediation into four assessable capabilities, from interpreting context to generating a norm-compliant response. A case study illustrates how those steps apply to a user’s information request.

Benchmark

  • The benchmark separates context understanding, expectation identification, appropriateness judgment, and response generation into four evaluation components.Context includes actors, information attributes, transmission principles, user intention, domain, and interaction traces; expectation identification selects relevant norms from alternatives.Source: Benchmark · pp. 2–3
  • Assessing these subproblems separately can reveal which skill constrains an assistant’s ability to protect user information.The paper notes that a deployed assistant need not perform the steps explicitly and could instead operate as a black box.Source: Benchmark · p. 3

Illustrating the Framework: A Case Study

  • In the doctor’s-office case study, the assistant identifies the requested information and relevant norm before judging whether sharing the user’s location is appropriate.The example distinguishes identifying location from an unrelated information attribute and selecting the norm relevant to appointment availability.Source: Benchmark · p. 3
  • The final response-generation component is intended to produce a reply consistent with the appropriateness judgment and relevant norms.The case study illustrates generating a response that shares information when appropriate and refuses when it is not.Source: Benchmark · p. 3

Dataset

CI-Bench uses dialogue and email scenarios to evaluate information-flow tasks in conversational settings. Its dataset construction combines structured scenario inputs with generated communications and test questions.

Dataset

  • Each test prompt contains a task scenario and may also include user information and expectations, which are unnecessary for context-understanding tasks.Each test case pairs an input prompt with an expected-output label.Source: Dataset · p. 3
  • 44,100 test cases span eight domains and are organized across dialogue and email formats and four benchmark phases.The dataset has a negative-to-positive label ratio of 7.4:1.Source: Dataset · p. 3
  • The pipeline builds structured scenarios from real-world dialogue characteristics, uses an LLM to generate task scenarios, and programmatically creates benchmark questions.Questions are presented with relevant background such as the task scenario, user data, or user expectations.Source: Dataset · pp. 3–4
  • Structured scenarios combine context, actors, information attributes, and transmission principles, with context including domain, user intention, and interaction traces.The design supports permutations of category values, while generated conversations are intended to convey the relevant CI components without irrelevant context switching.Source: Dataset · p. 4
  • Choice questions assess capabilities such as context understanding by asking the model to select a relevant context element from one correct and several incorrect options.Incorrect choices are randomly selected from other possible values of the feature.Source: Dataset · p. 4

Experiments

Across Gemini 1.0 sizes, assistants showed robust overall performance but room to improve on subtle information-flow distinctions. Smaller models struggled more, while appropriateness judgments improved with larger models and explicit norms.

Experiments

  • Gemini 1.0 assistants showed robust overall performance but room to improve on subtle information-flow distinctions; smaller models struggled, and well-defined guidelines significantly enhanced performance.Source: Experiments · p. 4

Understanding Context

  • Models handled information attributes and user intention relatively well but underperformed on recipients; AUC rose with model size, and dialogue/email format differences were not significant.Source: Understanding Context · p. 5

Identifying Relevant Norms

Judging Appropriateness

Generate Response

  • The final response task asks models for a short sentence that shares the user's information when appropriate and otherwise refuses the request.Source: Generate Response · p. 6

Discussion

Naive assistants understand contextual information better than they judge its appropriateness, while explicit data-flow rules improve judgment accuracy. CI-Bench is a first step toward studying assistant-driven information sharing, with limitations in labeling, context scope, and interaction assumptions.

Discussion

  • CI-Bench offers a first step toward understanding current capability gaps in assistant-driven information sharing and invites community contributions to refine the benchmark.The authors aim to support AI assistants aligned with users’ privacy expectations.Source: Discussion · p. 6

Experimental Takeaways

  • Top-tier language-model assistants understand contextual information but struggle to judge appropriateness, including when a data subject’s role shifts.They may also mistake normal information exchanges for inappropriate ones when they miss contextual subtleties.Source: Experimental Takeaways · p. 6
  • Well-defined data-flow rules and guidelines significantly improve models’ accuracy in assessing information exchanges.The authors identify this as a path for future enhancements.Source: Experimental Takeaways · p. 7

Limitations and Future Work

  • Manual labeling by two experts may introduce bias and inconsistency, reflecting the subjectivity of appropriateness judgments and the absence of universal ground-truth labels.The authors identify disagreements between experts as evidence of this subjectivity.Source: Limitations and Future Work · p. 7
  • Cultural norms and individual privacy preferences vary, motivating a more diverse rater pool and study of how personal and shared expectations can be reconciled.The paper notes regional, cultural, and sociopolitical variation in data-sensitivity norms.Source: Limitations and Future Work · p. 7
  • The benchmark lacks a justification mechanism and uses binary appropriate/inappropriate labels, leaving no way to represent uncertain or borderline judgments.The authors suggest Likert scales as one alternative for capturing more nuanced judgments.Source: Limitations and Future Work · p. 7
  • Samples are self-contained and stateless, with one context per session and no multiple contexts or context transitions.Future work could add multi-context sessions and extraneous information to exercise long-context models.Source: Limitations and Future Work · p. 7
  • Other boundaries include unverified role authenticity, difficulty disentangling sources of poor performance, and text-only data.The authors point to adversarial interactions, model and prompt variation, and multimodal data as areas for future study.Source: Limitations and Future Work · p. 7

Conclusion

CI-Bench evaluates AI assistants’ protection of personal information during inference using the Contextual Integrity framework and a scalable data-generation pipeline. Initial evaluations show promising zero-shot performance but challenges with nuanced scenarios, while explicit context-specific norms improve appropriateness judgments.

Conclusion

  • CI-Bench evaluates assistants’ ability to protect personal information during inference using the Contextual Integrity framework and a scalable data-generation pipeline.The benchmark spans domains and context features; initial evaluations found challenges with multiple topics and context switching, while explicit context-specific norms improved appropriateness judgments.Source: Conclusion · p. 7

Synthetic Data Generation

The dataset covers seven information categories and scenarios across eight domains. Its evaluation prompts take dialogue or email scenarios as input.

Synthetic Data Generation

  • The dataset includes 49 information pieces grouped into seven categories, from personal identifiers and demographics to health and other sensitive information.Each piece represents an attribute that could be disclosed in conversation.Source: Synthetic Data Generation · p. 9
  • The dataset uses 50 distinct scenarios spanning eight domains to cover varied real-world conversational contexts.The scenarios are intended to assess handling of sensitive information across situations.Source: Synthetic Data Generation · p. 9
  • Evaluation prompts take a dialogue or email scenario as input; the first two templates generate scenarios and the next four support the four core tasks.Scenario-generation parameters are specified in Table 2 of the paper.Source: Synthetic Data Generation · p. 9

Additional Experiment Results

Across the utility–privacy comparison, larger-model assistants perform better on both dimensions, and dialogues outperform emails. Additional analyses show category and domain differences in appropriateness judgment and low response-generation AUC across the evaluated models.

Additional Experiment Results

  • Evaluations used zero-shot responses, with Ultra at temperature 0.9 for open-ended tasks and greedy Gemini inference for choice-based tasks.Choice selections were based on the candidate with the highest assigned probability.Source: Additional Experiment Results · p. 9

Judgment by Domain and Information Category

  • Domain- and category-level analysis used specificity, defined as the probability of correctly classifying genuinely negative cases as not appropriate to share.The analysis was organized around both interaction domain and information category.Source: Judgment by Domain and Information… · p. 9
  • Demographic and financial information had higher specificity, while behavioral information challenged Gemini Ultra; email cases dipped slightly, with PII weaker in some domains.Healthcare and Finance were particularly demanding relative to Education and eCommerce in dialogue scenarios.Source: Judgment by Domain and Information… · p. 9, Figure 7 · p. 9, Figure 8 · p. 9

Generate Response

  • AUC values across all three models ranged from 0.3 to 0.5 for response generation, which requires context comprehension, norm identification, appropriateness judgment, and response formulation.Dialogue cases were more challenging than email cases across the three models, and performance showed a positive correlation with model size.Source: Generate Response · p. 10, Figure 9 · p. 10

Utility vs Privacy-Consciousness

  • Larger-model assistants perform better on both utility and privacy-consciousness, while dialogue cases outperform email cases.Utility is represented by sensitivity and privacy-consciousness by specificity in the two-dimensional comparison.Source: Figure 10 · p. 10, Utility vs Privacy-Consciousness · p. 10
Loading 2409.13903v1…