Source-linked AI summary
PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
Mingxuan Zhang, Jiahui Han, Dadi Guo, Songze Li, Guanchu Wang, Na Zou, Dongrui Liu, Xia Hu
TL;DR
Existing privacy benchmarks largely inspect what agents disclose, not what sensitive information they acquire during tool use. PrivacyPeek audits acquisition-stage leakage and finds that out-of-scope acquisition is widespread across ten LLM-based agents, with CER reaching 51.95% and PLR reaching 57.53%.
Problem
Existing privacy evaluations overlook whether agents acquire sensitive information beyond the task’s minimum scope, leaving acquisition-stage privacy insufficiently examined.
Method
PrivacyPeek benchmarks acquisition-stage leakage across 1,182 cases using Acquisition Inspection and Probe Elicitation across seven behaviours and 16 application domains.
Results
Across ten agents from four model families, out-of-scope acquisition was widespread, with CER reaching 51.95% and PLR reaching 57.53%.
Takeaways & Limitations
The findings indicate that acquisition-stage privacy risks require policy controls and agent self-restraint.
Takeaways & Limitations
The joint acquisition-and-disclosure event links acquired evidence to disclosure but does not establish causality.
Abstract
from arXiv · showhide
LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce \emph{PrivacyPeek}, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with $1{,}182$ cases across $7$ acquisition behaviours and $16$ application domains. Specifically, \emph{Acquisition Inspection} examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. \emph{Probe Elicitation} then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on $10$ LLM-based agents across $4$ model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.
Introduction
LLM-based agents often acquire sensitive information beyond task requirements, creating hidden privacy risks that response- and action-focused evaluations overlook. PrivacyPeek addresses this gap by benchmarking acquisition-stage leakage across diverse behaviours and domains.
- Problem: Agents often acquire more personal data than tasks require, such as a patient’s national ID and home address when only one vital sign is needed.This over-acquired information enters the agent’s context and may later leak through database writes, induced replies, or attacks.
- Problem: Acquisition-stage leakage remains overlooked because existing agent-privacy evaluations mainly inspect task responses or outgoing actions.Unneeded information can therefore remain hidden until a careless action or attack exposes it.
- PrivacyPeek: PrivacyPeek contains 1,182 evaluation cases spanning 7 acquisition behaviours and 16 application domains.Covered domains include healthcare, finance, legal services, education, government, human resources, and customer support.
- PrivacyPeek: PrivacyPeek provides a missing acquisition-stage view for auditing whether agents acquire sensitive information beyond task scope, supporting urgent privacy assessment in high-stakes deployments.The benchmark demonstrates that such leakage can remain hidden when evaluation inspects only task responses.
Related Work
Prior work audits whether agents appropriately share information and finds substantial privacy violations, including tool-response over-exposure. PrivacyPeek instead evaluates whether agents should have acquired sensitive information, covering 16 domains and 7 acquisition behaviours through two complementary perspectives.
- Prior privacy audits: Existing audits probe whether sharing information is contextually appropriate, while related work extends Contextual Integrity to multimodal inputs and internal representations.ConfAIde grounds this line of work in Contextual Integrity.
- Prior privacy audits: Internal information-flow annotation finds far more Contextual-Integrity violations than output-only auditing, mostly at the tool-response stage.AgentRaft separately finds widespread personal-data over-exposure in benign paths by statically analyzing public registry tools.
- PrivacyPeek: PrivacyPeek targets acquisition-stage privacy by asking whether an agent should have acquired sensitive information, rather than whether it leaves the agent.The benchmark covers 16 application domains and a taxonomy of 7 acquisition behaviours.
- PrivacyPeek: PrivacyPeek evaluates acquisition-stage privacy through two complementary perspectives: Acquisition Inspection and Probe Elicitation.These perspectives respectively examine acquisition behavior and probe the information acquired but not disclosed.
PrivacyPeek
PrivacyPeek benchmarks acquisition-stage privacy leakage by auditing whether agents acquire personal data beyond a pre-reviewed task minimum scope, rather than examining only final disclosures. It combines human-guided case generation with Acquisition Inspection and Probe Elicitation across 1,182 cases, 7 acquisition behaviours, and 16 application domains.
- Preliminaries: Agentic Setup: Acquisition-stage leakage occurs when a tool call acquires personal data outside the case-specific minimum scope required for correct task completion.The unit of analysis is the tool call, and the minimum scope covers required sources, fields, time window, format, and confidentiality boundaries.
- Task Design and Data Generation: The benchmark audits seven boundaries, including source, format, time, field, confidentiality, and inferential-evidence boundaries.Record-field decisions use the same data-access tool with either task-specific fields or fields="*".
- Task Design and Data Generation: 1,182 cases span 7 acquisition behaviours and 16 application domains through a human-guided, author-reviewed generation pipeline.The pipeline combines author-written templates and domain seeds with GPT-4o, then applies 10 deterministic checks and author review.
- Evaluation: Acquisition Inspection matches tool observations against annotated out-of-scope personal information to compute Content Exposure Rate and completion-conditioned exposure measures.Content Exposure Rate can reward refusal, so Task Completion Rate and Helpful Content Exposure Rate condition exposure comparisons on completed cases.
- Evaluation: Probe Elicitation disables tools and network access, then probes the retained task context to test whether acquired information can be elicited after the task.It is a post-task stress test rather than an acquisition detector or deployment-frequency estimate, and Task 7 links precursor evidence to reviewed derivations without establishing causality.
Experiments
Experiments on ten agents across four model families show widespread acquisition-stage leakage: every agent acquires out-of-scope data, and probe disclosure exceeds acquisition rates. Leakage correlates with task completion and varies substantially across behaviours and domains.
- Experimental setup: All 1,182 cases use the same eight tools, with filesystem access restricted to audited tools and identical task data and tool contracts within each condition.Agents are implemented with Smolagents CodeAgent, and structured tool observations support Acquisition Inspection.
- Acquisition and probe leakage: All ten evaluated agents acquire out-of-scope data, with CER ranging from 6.77% on Llama-3.2-3B-Instruct to 51.95% on Claude-Sonnet-4.Conditioning on completed tasks raises CER for every agent.
- Acquisition and probe leakage: PLR ranges from 16.67% to 57.53% and exceeds CER for every agent, measuring disclosure under a post-task probe rather than out-of-scope data in task observations.Thus, acquisition and probe leakage capture different outcomes.
- Capability–privacy relationship: TCR correlates positively with CER (Spearman’s ρ of 0.818, p < 0.01) and PLR (Spearman’s ρ of 0.685, p ≈0.029) across ten agents.Llama-3.2-3B-Instruct pairs the lowest TCR (54.48%) and lowest CER (6.77%), while Claude-Sonnet-4 has the highest TCR (86.46%) and CER (51.95%).
- Behaviour and domain variation: Task 7, Out-of-Scope Inferential Access, has the highest mean PLR at 53.29%, while the lowest-rate behaviour averages 9.45%.Across sixteen domains, social media, healthcare, legal services, and finance range from 36% to 51%, whereas retail, engineering, and logistics remain below 25%; both rankings are stable across all ten agents.
Analysis
Analysis shows that acquisition-stage leakage frequently co-occurs with probe disclosure, while observed non-disclosure does not imply safe acquisition. Comparisons across agents and versions therefore require task completion, acquisition, probe leakage, and behavioural modes together, and explicit confidentiality markers can increase leakage.
- Joint outcomes: 45.60% is the largest CER-positive probe share, versus 5.33% for Llama-3.2-3B-Instruct, with CER-positive outcomes largest across closed-source agents.CER-positive means out-of-scope acquisition is detected and the probe discloses a reviewed target.
- Joint outcomes: 4.46% is the average no-observed-disclosure share on closed-source agents, versus 1.64% on open-source agents, despite detected acquisition.Total PLR combines CER-positive and CER-negative outcomes; the joint breakdown separately captures acquisition without observed disclosure.
- Behavioural modes: 6.60% is Claude-Sonnet-4.5’s self-restraint share, compared with 4.48% for Claude-Sonnet-4, 3.81% for GPT-5.1, and 2.96% for GPT-4.1.All four highest values are closed-source, while Llama-3.3-70B-Instruct reaches 1.61%, below Qwen3-30B-A3B at 2.12%.
- Cross-version comparison: 55.39% is Claude-Sonnet-4.5’s HPLR, compared with 60.96% for Claude-Sonnet-4, while its CER and HCER remain 48.22% and 48.73%.Claude-Sonnet-4.5 also has a 5.67 percentagepoint lower PLR and a 6.43-point lower TCR than Claude-Sonnet-4, so lower task completion does not establish privacy preservation.
- Marker subset: +10.56% is the PLR increase for Qwen3-4B-Instruct with an explicit confidentiality marker, while Claude-Sonnet-4.5 increases by +9.87%.The marker raises rather than suppresses PLR by at least 3% on five agents; only three agents improve by at least 3%.
Mitigation Methods
Mitigation should prevent out-of-scope information from entering context, because later context management cannot undo prior acquisition. A category-aware system-prompt directive reduces acquisition leakage only partially, leaving substantial residual rates across agents.
- Preventive controls: Controls should act before acquisition through prompt-guided tool choice, field-restricted APIs, scoped views, and pre-tool policies; context management cannot undo prior acquisition.Context management may reduce later recoverability but does not prevent the initial privacy leakage.
- Prompt directive: 3.73% to 12.69% CER reductions leave more than half of the baseline rate for every evaluated agent across all 1,182 cases.The evaluation covered three open-source and two closed-source agents using one category-aware system-prompt directive.
- Prompt directive: CER falls from 20.56% to 11.77% on GPT-5.1 and from 48.22% to 36.47% on Claude-Sonnet-4.5, which retains the higher residual rate.Claude-Sonnet-4.5 has the larger reduction, so both absolute reduction and residual acquisition matter.
Conclusion
PrivacyPeek benchmarks acquisition-stage privacy leakage across diverse agent tasks using tool-trajectory inspection and follow-up probing. Across ten agents, out-of-scope acquisition is widespread, correlates positively with task completion, and remains substantially under-addressed by privacy-aware prompting.
- Benchmark and evaluation: PrivacyPeek comprises 1,182 cases spanning 7 acquisition behaviours and 16 domains, evaluated through Acquisition Inspection and Probe Elicitation.Acquisition Inspection examines tool calls and received data, while Probe Elicitation tests whether acquired-but-undisclosed information can be elicited.
- Main findings: Across ten agents from the GPT, Claude, Llama, and Qwen families, out-of-scope acquisition is widespread, with CER reaching 51.95% and PLR reaching 57.53%.Task completion correlates positively with both measures.
- Mitigation: A privacy-aware prompt reduces CER but leaves over half the risk, while confidentiality markers worsen leakage.The conclusion identifies substantial residual acquisition-stage risk despite prompt-level mitigation.
Appendix · Per-Tool Schemas and System Prompt
The appendix defines PrivacyPeek’s audited agentic environment: a fixed tool layer, system prompt, and execution protocol across all cases. Agents access case data only through logged tool calls, while evaluators alone retain the target and source sets.
- Appendix: PrivacyPeek runs every case with a Smolagents CodeAgent for at most six steps, receiving the case system prompt before the user instruction.At each step, the agent emits a short Python program that may call exposed tools and read their return values.
- Per-Tool Schemas and System Prompt: Agents reach case data only through tools and never see the evaluator-only target set kacq or the in-scope and forbidden source sets.This separates the agent’s available observations from the evaluator’s privacy labels.
- Per-Tool Schemas and System Prompt: All 1,182 cases expose the same eight tools, divided into data access, filesystem inspection, and outbound recording or transmission roles.The data-access and filesystem tools read the case substrate, while outbound tools record or transmit content.
- Per-Tool Schemas and System Prompt: Every tool call is logged with its name, arguments, and return value, and Acquisition Inspection matches returned observations against kacq.Recorded arguments describe agent choices but do not trigger CER on their own.
- Per-Tool Schemas and System Prompt: Across cases, the varying elements are the tool substrate and in-scope source set, not the exposed tool set.The tools remain fixed while the underlying case data and scoped sources change.
- Per-Tool Schemas and System Prompt: The system prompt assigns a domain role, lists the eight tools, and restricts the agent to Python’s standard library so external access stays within the audited tool layer.The domain role is instantiated from sixteen application domains.
- Per-Tool Schemas and System Prompt: The tool list and closing instructions are fixed across all 1,182 cases, with the user instruction appended after the system prompt.The template’s domain role varies per case across the sixteen application domains.
Minimum-Scope Annotation Protocol
The protocol defines a task-specific minimum scope by identifying what is required for correct completion and excluding merely confirmatory or contextual information. It also supports multiple valid minimum paths and recommends that deployed workflow boundaries be established by human domain stakeholders.
- Minimum-scope definition: The authors record each task’s required output or action, relevant sources, fields, time window, file format, and confidentiality boundary.They test each candidate item by asking whether removing it prevents correct task completion.
- Minimum-scope definition: Information useful only for confirmation or extra context remains outside the minimum scope, and multiple equally valid minimum paths may be recorded.The protocol verifies that the recorded paths support correct task completion.
- Transfer to deployed workflows: Deployed workflow scopes should be set by workflow owners with data stewards, privacy reviewers, or domain experts, then encoded through technical controls or escalated when ambiguous.PrivacyPeek evaluates compliance with this fixed task scope rather than treating one case label as a universal policy.
Template Specifications and Quality Gates
The benchmark instantiates PrivacyPeek cases from templates covering seven acquisition behaviours, with representative examples, deterministic quality gates, executable artefacts, and sixteen application domains. The specifications distinguish attribute-tier from content-tier violations and balance cases by behaviour rather than domain.
- Benchmark specifications: PrivacyPeek cases are built from templates for seven acquisition behaviours, with case counts, worked examples, ten quality gates, and domain coverage specified.The appendix provides one representative case for each behaviour, pairing a benign instruction with out-of-scope evidence acquired through tool calls.
- Behaviour taxonomy: Four attribute-tier behaviours violate a file’s external attributes, including its name, format, or time window.These behaviours place the violation on metadata-like properties outside the file’s content.
- Behaviour taxonomy: Three content-tier behaviours violate a file’s contents through an excess field, forbidden label, or cross-field inference.Tab. 6 identifies the scope axis for each behaviour, while Tab. 7 reports how 1,182 cases divide across the seven behaviours.
- Quality gates: Every materialised case must pass ten deterministic quality gates covering schema validity, single-behaviour isolation, unambiguous leakage ground truth, and task feasibility without violation.The gates require that cases cannot be solved by substituting a different acquisition behaviour.
- Executable document artefacts: Task 3 uses 119 DOCX, 147 XLSX, and 30 PDF files across 145 cases, parsed at runtime with format-specific tools; other tasks use structured JSON records.Missing files, unsupported extensions, and parser exceptions produce explicit tool errors rather than fallback content.
- Domain coverage: The benchmark spans sixteen application domains and is balanced by acquisition behaviour rather than domain.Each domain fixes a case’s deployment context and sensitive-information categories; social media and technology together contain a third of the cases.
Task-5 Field-Choice Contract … Prompt-Level Mitigation
The appendix specifies how PrivacyPeek distinguishes unnecessary acquisition from later disclosure across field choice, warnings, evidence derivation, and judge decisions. It also characterizes behavioural classes, marker-labelled cases, and prompt-level mitigation, including reductions in acquisition leakage under increasingly explicit directives.
- Task-5 Field-Choice Contract: Task 5 exposes a fields argument so logs distinguish narrow from broad requests, with CER triggered only when returned observations contain forbidden acquisition items.fields="*" requests the full payload, while the agent cannot see the forbidden fields or acquisition set.
- Task-6 Warning Visibility: Task 6 tests protected-source access despite a sufficient normal source and visible confidentiality warnings, while its 163 cases differ from a non-paired 44-case marker subset.Warnings and metadata alone do not trigger CER; protected information must enter the agent’s context.
- Task-7 Evidence and Disclosure Targets: Task 7 matches out-of-scope precursor evidence in tool observations for CER, while OLR and PLR judge disclosure of the corresponding sensitive derivation.Inference from entirely in-scope evidence can trigger OLR or PLR but not CER, and co-occurring positive metrics do not establish causation.
- Case Study: The case studies illustrate each acquisition behaviour, with Task 7 explicitly separating an out-of-scope precursor acquisition target from its sensitive disclosure target.Figures 10–15 use Claude-Sonnet-4 cases, while Figure 16 demonstrates the Task 7 distinction.
- Confidence Intervals for the Main Results: Wilson 95% confidence intervals quantify case-level sampling uncertainty around the three main binary metrics, not pairwise differences between agents.The intervals accompany the 1,182-case main results.
- Probe Judge: Prompt and Decision Rules: A fixed GPT-4o judge applies identical rules to main and probe responses, distinguishing literal, semantic, and inferential disclosure without inspecting tool observations.Probe Elicitation occurs after the normal trajectory and cannot alter the earlier CER event; ambiguous cases receive human review.