PaperlayerView exact revision on arXiv

Source-linked AI summary

Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?

Anmol Goel, Iryna Gurevych

arXiv:2606.23189v1cs.AIcs.CL

TL;DR

Computer-use agents can expose personal information from one application in another context, but existing evaluations provide limited evidence about this risk. The paper introduces AgentCIBench to test contextual disclosure across realistic computer-use scenarios and finds that task-completion ability does not reliably imply safe information handling.

  • Problem

    Existing privacy evaluations under-explore contextual information flow during multi-application, multi-turn computer use, limiting evidence about whether agents share only contextually appropriate information.

  • Method

    AgentCIBench evaluates computer-use agents in application-state scenarios with task-specific must-share and must-not-share items, using automated stress-test generation across three contextual-integrity failure modes.

  • Results

    Task-completion rankings do not transfer to safety: several of the strongest task-performing agents are among the leakiest, with nearly uncorrelated task and disclosure orderings.

  • Takeaways & Limitations

    Disclosure should be reported as a separate evaluation axis when benchmark rankings guide deployment on sensitive personal state.

  • Takeaways & Limitations

    Absolute leakage rates are not estimates of real-world user harm because the benchmark uses a controlled environment and an intentionally stress-tested scenario pool.

Abstract

from arXiv · show

Computer-use agents (CUAs) now act on a user's behalf across personal applications such as email, calendars, and to-do lists. This cross-application access is useful, but it also creates a privacy risk that has been largely overlooked: when an agent works in one context, it can pull in information from another that is inappropriate in that context. Hence, we introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios. We target three common failure modes in CUAs: visual co-location, where the agent pulls in prohibited items that sit next to the task target in the UI; task-ambiguity overshare, where the agent dumps dense personal state in response to an under-specified prompt; and recipient misalignment, where the agent sends content to an addressee for whom it is inappropriate. We evaluate 15 frontier agents and find a surprisingly high failure rate: 11 of 15 leak on more than 50% of scenarios, with an average leakage of 67.9%, and the same failures persist when agents act end-to-end in the environment to complete the task. We release AgentCIBench to encourage the development of safer computer-use agents and position contextual disclosure testing as a pre-deployment safety check.

1 Introduction

Computer-use agents can access information across personal applications, but contextual disclosure depends on whether content is appropriate for the task and recipient. AgentCIBench evaluates this risk through scenario-specific utility and leakage judgments, revealing persistent leakage and motivating lightweight mitigations.

  • Motivation: Cross-application access helps CUAs complete tasks but can expose personal calendar entries or private notes that are inappropriate for the intended recipient.CUAs may gather information from inboxes, calendars, to-do lists, and notes before producing externally visible messages or events.
  • Evaluation lens: Contextual integrity evaluates whether information flows respect the norms of its context, including the appropriate actor, recipient, content, and transmission principle.The same information may be appropriate for one recipient, such as a family group, but inappropriate for another, such as a colleague.
  • Benchmark: AGENTCIBENCH specifies application state, task, and recipient, then scores externally visible outputs against must-share utility items and must-not-share leakage items.An automated scenario-surfacing engine generates realistic stress-test scenarios at scale.
  • Empirical findings: 15 frontier CUAs show that high task completion does not imply low leakage, with the strongest task performers sometimes among the worst disclosure offenders.The study reports distinct alignment patterns across disclosure modes.
  • Deployment and mitigations: Leakage persists, and sometimes increases, during end-to-end evaluation in the multi-application environment rather than final-disclosure-only evaluation.Three lightweight interventions reduce engagement-conditioned leakage by 33 to 36 percentage points while improving utility across all failure modes.

2 Related Work · 3 Studying CUA Disclosure

The paper frames AgentCIBench as a contextual-integrity evaluation for cooperative computer-use agents, addressing gaps in multi-application, persistent-state, normatively scored disclosure testing. It operationalizes disclosure as selecting required information while excluding contextually inappropriate visible state across three CI-grounded failure modes.

  • 2 Related Work: Contextual integrity defines privacy as appropriate information flow governed by recipient, form, and purpose, rather than by sensitivity alone.Prior work operationalizes these norms through crowdsourcing and language-model benchmarks.
  • 2 Related Work: AgentCIBench differs from adversarial safety benchmarks by testing unintentional contextual-integrity violations from cooperative agents whose outputs over-include information relative to the task context.Capability benchmarks measure task completion, whereas this evaluation measures whether transmitted information is contextually appropriate.
  • 2 Related Work: Existing privacy evaluations under-explore multi-application use, persistent multi-turn UI state, and normative information-flow scoring under contextual integrity.AgentCIBench targets their intersection with cooperative agents acting in rendered applications that must share some information while withholding other information.
  • 3 Studying CUA Disclosure: The evaluation presents application state and a task prompt, then tests whether the agent’s output includes required items while excluding prohibited visible items.The framework covers externally visible actions such as sent messages, saved calendar events, and posted notes or replies.
  • 3 Studying CUA Disclosure: Utility, leakage, refusal, and engagement-conditioned leakage quantify disclosure outcomes, separating appropriate content selection from low leakage achieved by refusing to act.A deterministic matcher and LLM judge jointly detect exact, near-exact, and paraphrased information mentions.
  • 3.1 A taxonomy of CUA disclosure failures: Three CI-grounded failure modes target surfaced attributes, transmission principles, and recipients: visual co-location, task-ambiguity overshare, and recipient misalignment.Together, they cover disclosure decisions involved in summarizing, forwarding, and replying on a user’s behalf.
  • 3.1 A taxonomy of CUA disclosure failures: Visual co-location tests spatial filtering, task-ambiguity overshare tests whether underspecified requests trigger indiscriminate dumping, and recipient misalignment tests recipient-conditioned sharing.These modes can overlap, but each scenario receives one primary authored failure mode; the released pool contains 75 TAO, 24 RMA, and 18 VCL scenarios.
  • 3.1 A taxonomy of CUA disclosure failures: State-grounded evaluation holds scenario state constant across models, isolating disclosure choices from click accuracy, page loading, and tool-use failures.The same scenario JSON enables paired comparisons across models, defenses, and failure modes.

4 The AGENTCIBENCH Harness

AGENTCIBENCH is a generative harness for testing contextual-integrity failures in computer-use agents across realistic multi-application tasks. It combines MCTS-based scenario generation, an OpenApps workspace, hybrid deterministic/LLM scoring, and a re-runnable released pipeline.

  • Harness components: AGENTCIBENCH combines a scenario-surfacing engine, the OpenApps workspace renderer, and a hybrid scoring pipeline.The engine generates cross-app tasks containing information to share and withhold; OpenApps renders them as personal multi-app workspaces, and scoring checks task completion and leakage.
  • Scenario-surfacing engine: MCTS searches for high-utility disclosure scenarios by mutating ordinary personal-assistant task seeds across three contextual-integrity failure modes.The search represents complete scenarios and starts from seeds such as workplace updates, calendar sharing, shopping, and procurement.
  • Scenario-surfacing engine: Proxy-agent reward requires both task completion and inappropriate disclosure, while a novelty bonus and duplicate filtering diversify the scenario pool.Task refusal or failure receives zero reward; the 1–5 severity score guides search, whereas headline results use binary leakage.
  • OpenApps environment: OpenApps instantiates scenarios in a six-app personal workspace containing Messenger, Calendar, Maps, ToDo, Code Editor, and Shop.The live multi-tab UI includes both shareable and leakable information, and the state-grounded setting supplies structured state before direct action-JSON output.
  • Hybrid scoring: Hybrid scoring merges a deterministic matcher with an LLM judge, counting a leak when either detects it under the pipeline’s conservative textual-validation rule.The matcher uses normalized containment, token coverage, and sequence similarity; the judge identifies task completion, leaked items, and contextual-integrity severity.
  • Curation and release: The released artifact is a re-runnable generation pipeline whose scenarios undergo automated reward filtering, duplicate removal, and manual coherence checks.The pool can be regenerated or extended as proxy models and frontier agents change; incoherent scenarios are discarded.

5 Experimental Setup

The experimental setup evaluates fifteen proprietary and open-weight agents across 117 paired scenarios per agent, using hybrid scoring to measure utility, leakage, refusal, and engagement-conditioned leakage.

  • Agents: 15 agents span proprietary and open-weight families, including Claude, GPT, Gemini, Grok, Qwen, Kimi, DeepSeek, MiniMax, Gemma, and GLM variants.Appendix D lists all evaluated models.
  • Scoring: Hybrid scoring merges deterministic matches with LLM-judge detections into a single leak set and reports utility U, leakage L, refusal rate, and engagement-conditioned leakage Leng.Matcher thresholds, judge prompts, and judge–matcher agreement analyses are provided in Appendix E.
  • Sample size: 117 paired scenarios per agent support aggregate comparisons of leakage and utility rather than exhaustive coverage of possible computer-use tasks.The main-text effects exceed the minimum detectable differences implied by these sample sizes; power analyses and sensitivity checks appear in Appendix F.

6 Do CUAs Follow Contextual Integrity? · 7 Do Disclosures Persist in End-to-End UI Interactions?

Across 15 agents evaluated on everyday AgentCIBench scenarios, disclosure leakage is widespread, poorly predicted by utility, and confounded by refusal behavior. In end-to-end UI execution, substantial leakage reappears for both tested agents on engaged trajectories.

  • 6 Do CUAs Follow Contextual Integrity?: 15 agents were evaluated on AgentCIBench scenarios spanning three contextual-integrity failure modes and derived from 28 hand-authored everyday personal-assistant requests.The evaluation reports utility, raw leakage, refusal, and engagement-conditioned leakage.
  • 6 Do CUAs Follow Contextual Integrity?: 12 of 15 agents leak on more than half of scenarios, six exceed 80.0% leakage, and average utility and leakage are 68.8% and 67.9%, respectively.Disclosure violations occur nearly as often as successful task completion, rather than appearing only as rare failures.
  • 6 Do CUAs Follow Contextual Integrity?: Among agents above 75.0% utility, engagement-conditioned leakage spans 14.0% to 98.3%, an 84-point spread despite broadly successful task completion.Gemini-3.1-Pro and Gemini-3-Flash combine the two highest utilities, 96.6% and 88.0%, with engagement-conditioned leakage of 98.3% and 94.0%.
  • 6 Do CUAs Follow Contextual Integrity?: Raw leakage can conceal refusal: Claude-Opus-4.7 has L=13.7% and Leng=14.0% with 1.7% refusal, whereas GPT-5.4 has L=18.8% but Leng=32.4% with 41.9% refusal.For Kimi-K2.6, the same distinction is 62.4% raw leakage versus 80.2% engagement-conditioned leakage, motivating engagement-conditioned leakage as the primary disclosure metric.
  • 6 Do CUAs Follow Contextual Integrity?: Safety-tuned models keep TAO and RMA low but remain elevated on VCL: Claude-Opus-4.7 scores 9.3% TAO, 12.5% RMA, and 33.3% VCL.This is a 24-point gap, consistent with stronger handling of recipient and task-scope distinctions than visually adjacent prohibited content.
  • 7 Do Disclosures Persist in End-to-End UI Interactions?: The end-to-end UI study deploys Claude-Opus-4.7 and Claude-Sonnet-4.6 on a 50-scenario stratified set in the rendered OpenApps environment using mixed screenshot and accessibility-tree access.It tests whether disclosure patterns from structured state persist when agents navigate the interactive UI and emit final task outputs.
  • 7 Do Disclosures Persist in End-to-End UI Interactions?: On engaged UI trajectories, Claude-Sonnet-4.6 leaks on 8/10 runs (Leng=80.0%; state-grounded: 54.5%), while Claude-Opus-4.7 leaks on 6/14 (Leng=42.9%; state-grounded: 14.0%).Thus, substantial leakage reappears for both agents when they successfully emit an artifact through end-to-end interaction.

Model VCL (%) TAO (%) RMA (%) · 8 Can Disclosure Be Mitigated? · 9 Discussion and Policy Implications

Leakage varies by failure mode and model alignment, while live-UI execution preserves substantial disclosure risk. Three prompt-only defenses reduce leakage across models and modes while improving utility, supporting contextual-integrity evaluation as a separate pre-deployment safety axis.

  • Model VCL (%) TAO (%) RMA (%): Safety-tuned models keep TAO and RMA low while elevating VCL; Kimi suppresses VCL and RMA but spikes on TAO, whereas weakly aligned models saturate every mode.These engagement-conditioned patterns distinguish model behavior by failure mode rather than task capability alone.
  • Model VCL (%) TAO (%) RMA (%): 42.9% for Opus and 80.0% for Sonnet: completed live-UI runs show engagement-conditioned leakage at or above state-grounded baselines.Many nonengaged runs are budget-truncated before emitting an artifact rather than refused, creating a navigation bottleneck.
  • 8 Can Disclosure Be Mitigated?: Three prompt-only defenses target disclosure through restrictive scoping, a four-point contextual-integrity rubric, or explicit recipient and norm identification.The defenses are tested on Claude-Opus-4.7, GPT-5.4, and DeepSeek-v4-Pro without retraining or tool changes.
  • 8 Can Disclosure Be Mitigated?: 51.7% to 16.2%: recipient-typed cuts average engaged leakage, compared with 15.8% under rubric-informed and 19.0% under restrictive defenses.All three defenses reduce engaged leakage by 33 to 36 points, and each lowers leakage for every tested model relative to its undefended baseline.
  • 8 Can Disclosure Be Mitigated?: 15.7 to 23.1 percentage points: average utility rises under all three defenses, with recipient-typed increasing mean utility from 63.2% to 86.3%.Recipient-typed and rubric-informed prompts lead agents to identify the recipient and permissible content before writing.
  • 9 Discussion and Policy Implications: 12 of 15 frontier agents fail on more than half of realistic scenarios, while task-completion rankings nearly uncorrelate with safety and several high-utility agents are among the leakiest.The paper recommends contextual-integrity evaluation as a separate disclosure axis and proposes rerunnable benchmarking plus prompt defenses in post-training pipelines.
  • 9 Discussion and Policy Implications: 13 steps: Claude-Opus-4.7 cycles between OpenTodos and Nora’s thread, then sends all 12 to-do items, including four inappropriate items.The four shared items are plumber, vet, dry-cleaner, and an external Stripe contact.

10 Conclusion · Limitations · Ethical Considerations

AGENTCIBENCH evaluates whether computer-use agents make context-appropriate disclosure decisions across sensitive personal applications, finding frequent leakage and persistent end-to-end failures. It also demonstrates steerable mitigation, while framing results as controlled stress-test evidence and releasing the benchmark for safer pre-deployment evaluation.

  • 10 Conclusion: AGENTCIBENCH addresses a gap in evaluations that mostly measure task completion rather than context-appropriate disclosure across sensitive personal state.It measures the final disclosure decision of computer-use agents under contextual-integrity constraints.
  • 10 Conclusion: 11 of 15 agents leak on more than half of scenarios, with several high-utility agents among the leakiest.The benchmark evaluates fifteen frontier and open-weight agents.
  • 10 Conclusion: End-to-end rendered-UI artifacts show the same disclosure pattern, including for the two agents with lowest state-grounded engaged leakage.This indicates that the observed failures persist when agents operate through the environment rather than only through isolated evaluation outputs.
  • 10 Conclusion: Mitigations reduce engaged leakage by 33–36 points while raising utility by 16–23 points, suggesting steerability without retraining.AGENTCIBENCH is released as a re-runnable benchmark for CI stress-testing of computer-use agents.
  • Limitations: OpenApps is a controlled six-app workspace, and its absolute rates should be interpreted as relative orderings because scenarios are adversarially generated.The scenario pool is intentionally harder than a random sample of agent traffic.
  • Limitations: The end-to-end study covers two agents on a 50-scenario stratified subset, so engaged-leakage confidence intervals are wide and deployment results are suggestive rather than precise.The subset comes from the same adversarial scenario-engine pool.
  • Ethical Considerations: Because the benchmark targets privacy failures, it could be misused to elicit or optimize leakage, so it uses synthetic workspaces, scores specified units, and avoids attack-oriented framing.Released scenarios are intended for pre-deployment evaluation, regression testing, and mitigation development.
  • Ethical Considerations: Absolute leakage rates are not estimates of real-world user harm; comparisons should emphasize relative behavior, paired results, and failure modes, while labels may vary across cultures and preferences.OpenApps is controlled and intentionally stress tested, and contextual-integrity judgments may differ across organizations and users.

A Seed scenarios · B Scenario-surfacing engine · C OpenApps environment

The benchmark constructs 117 scenarios from seeded disclosure-error patterns using an MCTS surfacing engine, then evaluates agents in a six-app OpenApps workspace through state-grounded and end-to-end tracks. The released scenarios and scoring pipeline combine targeted mutations, deterministic filtering, and artifact-based disclosure assessment.

  • A Seed scenarios: 36 distinct seeds instantiate three disclosure-error modes from deployed-CUA demonstrations, cross-app user-study patterns, and conversational contextual-integrity failures.The full seed set, JSON schemas, and seed-to-source mapping are released.
  • A Seed scenarios: 117 evaluation prompts derive from 28 seed templates, with 1–10 mutated variants per seed and coverage across seven thematic categories.Workplace status reporting contributes 38/117 scenarios (32.5%), while calendar-availability sharing contributes 23/117 (19.7%).
  • B Scenario-surfacing engine: Per-mode rates are lower bounds because scenarios receive one targeted label even though visual co-location, task ambiguity, and recipient misalignment can overlap.The labels identify the mutator’s optimization target rather than mutually exclusive failure categories.
  • B Scenario-surfacing engine: Three CI-targeted mutation strategies map to visual co-location, task-ambiguity overshare, and recipient misalignment.Semantic_Entanglement creates thematic UI adjacency, Ambiguity_Trap underspecifies prompts, and Identity_Bleed reframes recipients or requests output for another recipient.
  • B Scenario-surfacing engine: R⋆=4.0 on the 0–5 scale is the keep threshold, requiring high utility with leakage severity of at least 4 on the majority of three proxy agents.The per-candidate reward is utility multiplied by contextual-integrity violation severity and averaged across proxies.
  • B Scenario-surfacing engine: 36 seeds produced 117 released scenarios after retaining 480 high-reward candidates, selecting a 140-scenario pre-dedup pool, and removing 7 exact and 16 near duplicates.Exact removals were 5.0% of the pre-dedup pool; near-duplicate removals were 11.4%.
  • C OpenApps environment: OpenApps is a six-app BrowserGym workspace containing a to-do list, messenger, calendar, maps client, shop, and code editor, instantiated per scenario.Each scenario specifies the app subset and initial content populated before agent invocation.
  • C OpenApps environment: The same scenario JSON supports state-grounded one-inference evaluation and end-to-end visual interaction capped at 20 actions, with disclosure scored from final artifacts.End-to-end results use mixed screenshot and accessibility-tree access, and scoring targets sent messages, saved events, or other emitted artifacts rather than intermediate observations.

D Evaluated models · E Judge and matcher details · F Power analysis

The study evaluates fifteen temperature-0 agents and combines deterministic matching with LLM judging to score utility and leakage. Power analyses indicate that the study detects specified leakage-rate differences, while binary outcomes are stable across scoring tracks despite substantial severity disagreement.

  • D Evaluated models: Fifteen agents are evaluated at temperature 0, using official provider APIs for proprietary models and local vLLM serving for open-weight models.The evaluated models and their developers are listed in Table 7.
  • E Judge and matcher details: The deterministic matcher compares shared content with Vshare and Vleak using normalized containment, token coverage, and sequence similarity tests.Leak matches generally require at least 4 matched tokens, at least 72% token coverage, and at least 88% sequence similarity, with stricter handling for short items.
  • E Judge and matcher details: Final utility requires task completion and support for every shared item, while the final leak set unions deterministic leaks with strongly supported LLM-claimed leaks mapped to Vleak.This hybridization is designed to guard against false positives from generic vocabulary; headline tables use binary leakage, while severity uses a 1–5 scale.
  • E Judge and matcher details: 84.8% of the 1,755 model-scenario cells have matching binary utility outcomes across deterministic and LLM-judge tracks.The tracks disagree on 266 cells, while the merged leak set is at least as conservative as the deterministic track.
  • E Judge and matcher details: 57.9% of cells disagree on ordinal severity, with a mean absolute reward gap of 1.29 on the 0–5 scale.This reflects the deterministic track assigning severity 4 or 5 to any non-empty leak set, whereas the LLM judge distributes severity more smoothly.
  • F Power analysis: At n=117, the main study resolves leakage-rate differences of 11.7 percentage points at 80% power and 14.8 percentage points at 95% power.The two-sided two-proportion z-test uses α=0.05; the deployment study has n=50.
  • F Power analysis: At n=50, the deployment study resolves leakage-rate differences of 17.2 percentage points at 80% power and 21.4 percentage points at 95% power.Per-failure-mode subgroups use the same parameter choices but smaller sample sizes, including TAO n=75 and RMA n=24.
  • F Power analysis: Retrospective power exceeds 0.997 for every reported comparison, while the smallest end-to-end arm has the widest confidence intervals and engaged-run counts of 14 and 10.The paper flags these denominators because the direction of the deployment shift should not be over-interpreted.

G State-grounded confidence intervals … M End-to-end transfer: trajectory analysis

The evaluation reports leakage with confidence intervals, refusal-aware comparisons, capability–disclosure rank inversions, mode-specific breakdowns, and behavior decompositions. Defense and end-to-end analyses show that refusal, navigation limits, and live interaction materially shape observed leakage without eliminating persistent vulnerabilities.

  • G State-grounded confidence intervals: Table 10 reports per-model raw leakage with 95% bootstrap confidence intervals, refusal rate, and engagement-conditioned leakage.The intervals use 10,000 resamples and the percentile method.
  • H Raw vs. engagement-conditioned leakage: The vertical gap between raw and engagement-conditioned leakage measures refusal, with near-diagonal agents showing restraint and above-diagonal agents masking leakage through refusal.Claude-Opus-4.7 has 1.7% refusal, while GPT-5.4 has 41.9% refusal masking leakage.
  • I Capability vs. disclosure ranks: The two highest-utility agents, Gemini-3.1-Pro and Gemini-3-Flash, rank at the bottom of disclosure, while Claude-Opus-4.7 leads disclosure despite dropping utility positions.Crossings between utility and disclosure rankings indicate utility-versus-disclosure inversions.
  • J Per-mode breakdown: Engagement-conditioned leakage is reported separately for visual co-location, task-ambiguity overshare, and recipient misalignment.The corresponding scenario counts are VCL n=18, TAO n=75, and RMA n=24.
  • K Behaviour decomposition: Behavior decomposition distinguishes completed clean, completed leak, incomplete clean, and incomplete leak outcomes across every model–scenario cell.The incomplete-leak segment tests whether low raw leakage reflects genuine safety or refusal-driven noncompletion.
  • K Behaviour decomposition: Kimi-K2.6 has 34.2% incomplete clean and 22.2% incomplete leak, while GPT-5.4 has 50.4% incomplete clean and 5.1% incomplete leak.These figures show different tradeoffs between refusal-related noncompletion and leakage.
  • L Defense sweep details: The defense sweep evaluates none, restrictive, rubric_informed, and recipient_typed conditions on Claude-Opus-4.7, GPT-5.4, and DeepSeek-v4-Pro across the full 117-scenario set.Each defense–agent cell contains 117 records, yielding 351 records per defense and 1,404 total.
  • M End-to-end transfer: trajectory analysis: End-to-end leakage exceeds state-grounded baselines for both models, despite severe step-budget truncation and small, engagement-biased samples.With a 20-step budget, 99 of 100 runs were truncated; Opus leaked on 6 of 14 engaged runs (42.9%) versus 14.0% state-grounded, while Sonnet leaked on 8 of 10 (80.0%) versus 54.5%. Four of seven Opus leaks and six of eleven Sonnet leaks also appeared state-grounded.

N Comparison with prior CI and agent-privacy benchmarks

AgentCIBench extends prior contextual-integrity and agent-privacy benchmarks by combining multi-app personal state, rendered-UI evaluation, adversarial generation, reproducibility, cooperative-agent violation testing, and cross-family transfer. Earlier benchmarks cover narrower settings, while AgentCIBench’s combined design exposes distinct disclosure failures that omitting any one property could mask.

  • Prior benchmark scope: Prior benchmarks largely use text-only, single-app, single-turn prompts, while AgentDAM adds partial UI and AgentDojo adds multi-app agents under adversarial prompt injection.ConfAIde, PrivacyLens, and CI-Bench rely on hand-authored or template-instantiated scenarios; AgentDAM remains single-app and non-adversarial, and AgentDojo lacks a re-runnable generator.
  • AgentCIBench’s distinguishing design: AgentCIBench is the only benchmark combining six properties: multi-app personal state, rendered UI, adversarial generation, a re-runnable engine, cooperative-agent violation testing, and cross-family transfer evidence.Its engine scales to new agents and explicitly tests unintentional contextual-integrity violations by a cooperative agent.
  • Why the combination matters: Removing any one of these properties would mask a different class of disclosure failures observed in the paper’s evaluation sections.The benchmark argues that the properties matter jointly rather than independently.

O Qualitative examples (state-grounded) … Q.1 Engine prompts

The appendix presents qualitative state-grounded examples, documents AI-assistant usage, and reproduces the prompts used for AgentCIBench’s generation, evaluation, judging, and defense sweeps.

  • O Qualitative examples (state-grounded): Three different agents produced leaks in state-grounded evaluations, while two Claude-Opus-4.7 completions on the same scenarios did not leak.The examples report each scenario prompt, ground-truth must-share and must-not-share sets, verbatim shared_content, and the LLM judge’s explanation.
  • O Qualitative examples (state-grounded): The qualitative scenarios were drawn unchanged from the released pool, and each example pairs the prompt with the ground-truth sharing requirements and the agent’s emitted content.
  • O Qualitative examples (state-grounded): Claude-Opus-4.7 completed the same auth-debug and calendar-availability scenarios without leaking, showing the failures were not forced by those scenarios.
  • P Usage of AI Assistants: Claude 4.7 Opus and GPT 5.5 were used for coding-related tasks, while GPT-5 and Claude Sonnet supported light editing.Light editing included re-wording, grammar, and proofchecking for the paper.
  • Q Prompts (verbatim): The appendix reproduces verbatim every AGENTCIBENCH prompt used for scenario generation, evaluation, judging, and defense sweeps.These include engine-side prompts in §Q.1 and defense-side prompts prepended to the agent system message in §Q.2.
  • Q.1 Engine prompts: The Q.1 engine prompts cover mutation, per-strategy instructions, proxy behavior, and judging.Each is reproduced verbatim in Figures 21–24.

Q.2 Defense prompts

Q.2 describes three contextual-integrity defense prompts that are prepended to the agent’s system prompt before scenario inference, with a no-defense baseline using the default system prompt. The prompts include restrictive, rubric-informed, and recipient-typed variants that require selective handling of outgoing content.

  • Defense setup: Three defense prompts are loaded into the system message before scenario inference, while the no-defense baseline leaves the default system prompt unchanged.The defenses are applied by prepending their text to the system prompt.
  • Restrictive defense: The restrictive defense instructs the agent to be deliberately selective about what it reads and reproduces.It frames the agent as operating across apps the user owns while handling the user’s entire screen.
  • Rubric-informed defense: The rubric-informed defense requires scoring and revising every outgoing artifact until it passes contextual-integrity checks.The rule applies to messages, calendar events, notes, shop orders, and file saves.
  • Recipient-typed defense: The third variant is a recipient-typed defense prompt intended to tailor disclosure decisions to the addressee.The supplied passage identifies this variant but does not reproduce its full text.
Loading 2606.23189v1…