PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents

Shouju Wang, Fenglin Yu, Xirui Liu, Xiaoting Qin, Jue Zhang, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan

arXiv:2509.17488v1cs.CRcs.AI

TL;DR

LLM privacy research has a gap between recognizing private information and controlling its disclosure in agent workflows, while existing benchmarks are largely static. The paper introduces PrivacyChecker, a contextual-integrity mitigation framework, and PrivacyLens-Live for dynamic MCP and A2A evaluations; reported leakage falls substantially while task helpfulness is preserved.

  • Problem

    Most privacy benchmarks are static and single-agent, and models can recognize private information yet still leak it during generation.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    PrivacyChecker uses contextual-integrity reasoning to identify and evaluate information flows and guide agent behavior at inference time.

    Source: §1 · p. 1

  • Results

    PrivacyChecker reduces privacy leakage by over 75% across diverse LLMs, including GPT-4o’s 33.06% to 8.32% and Qwen3-14B-NT’s 37.52% to 15.91%.

    Source: §1 · p. 2, §3.3.1 · p. 5

  • Takeaways & Limitations

    PrivacyLens-Live reveals substantially higher leakage risks than prior single-agent probes in realistic MCP and A2A workflows.

    Source: §5 · p. 9

    PrivacyLens-Live currently supports a limited number of tool integrations, and its MCP and A2A instantiations may evolve.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The increasing autonomy of LLM agents in handling sensitive communications, accelerated by Model Context Protocol (MCP) and Agent-to-Agent (A2A) frameworks, creates urgent privacy challenges. While recent work reveals significant gaps between LLMs' privacy Q&A performance and their agent behavior, existing benchmarks remain limited to static, simplified scenarios. We present PrivacyChecker, a model-agnostic, contextual integrity based mitigation approach that effectively reduces privacy leakage from 36.08% to 7.30% on DeepSeek-R1 and from 33.06% to 8.32% on GPT-4o, all while preserving task helpfulness. We also introduce PrivacyLens-Live, transforming static benchmarks into dynamic MCP and A2A environments that reveal substantially higher privacy risks in practical. Our modular mitigation approach integrates seamlessly into agent protocols through three deployment strategies, providing practical privacy protection for the emerging agentic ecosystem. Our data and code will be made available at https://aka.ms/privacy_in_action.

1 Introduction

LLM agents increasingly handle sensitive information through tools and inter-agent protocols, but privacy evaluation and mitigation have not kept pace with their behavior in realistic workflows. The paper introduces PrivacyChecker to address the judgment–action gap and PrivacyLens-Live to evaluate privacy in dynamic MCP and A2A settings.

1 Introduction

  • Existing privacy benchmarks are mostly static and single-agent, relying on fixed probes or scripted dialogues that miss open-ended agent behavior.The authors note that simulated dialogues can also overlook evolving agent chains and inter-agent protocols.Source: §1 · p. 1
  • PrivacyChecker reduces privacy leakage by over 75% across diverse LLMs without task performance loss.It is a model-agnostic, inference-time framework based on Contextual Integrity.Source: §1 · p. 2
  • PrivacyChecker addresses the gap between agents’ ability to recognize sensitive information and their failure to act accordingly during generation, especially in multi-step workflows.Source: §1 · pp. 1–2
  • PrivacyLens-Live transforms static benchmarks into dynamic, multiagent evaluations using MCP and A2A protocols.The paper evaluates mitigation effectiveness, compares static and live evaluations, and explores integration strategies.Source: §1 · pp. 1–2

2 Related Work

Prior work addresses privacy through training-stage protections, inference-time instructions, and contextual-integrity approaches, but it leaves open how privacy judgments translate into agent actions and how privacy risks appear in live, multiagent workflows.

2.1 Privacy Preservation for LLMs

  • Training-stage privacy work includes Differential Privacy, Federated Learning, and CPPLM, while inference-time work includes privacy instructions and contextual-integrity approaches.AirGapAgent addresses context-hijacking, and a CI-based supervisor focuses on single-key form filling.Source: §2.1 · p. 2
  • PrivacyChecker extends contextual-integrity mitigation to multi-step agent protocols and diverse tasks and interaction patterns.It identifies information flows, reasons over them, and dynamically steers agent behavior at inference time.Source: §2.1 · p. 2

2.2 Gap Between LLM Judgment and Action

  • Privacy benchmarks show that models can recognize private information yet still leak it during generation, a gap not specifically addressed by prior privacy-preservation efforts.Prior consistency-focused mitigation has shown promise in Q&A contexts.Source: §2.2 · p. 2

2.3 Evaluating LLMs’ Privacy Risk

  • Existing privacy benchmarks are static and single-agent, leaving MCP- and A2A-based workflow risks unmeasured.PrivacyLens-Live builds on PrivacyLens to evaluate and mitigate privacy risk in dynamic, multiagent interactions.Source: §2.3 · p. 3
  • State-of-the-art LLMs leak private information in over 25% of cases even with privacy-aware prompts [Shao et al., 2024].PrivacyLens-Live is presented as a dynamic extension of the static benchmark setting.Source: §2.3 · p. 3

3 PrivacyChecker

The section identifies a privacy judgment–action gap in agent generation, then introduces PrivacyChecker’s contextual-integrity flow checks and evaluates them on action-based tasks, including ablations.

3.1 Understand the Privacy Judge–Action Gap

  • 98.0% of the time the model classifies private information as sensitive, yet it still leaks private details in 33.1% of cases.Manual analysis found that leaked-instance reasoning rarely included privacy considerations and often focused only on task completion, motivating explicit enforcement during generation.Source: §3.1 · p. 3

3.2 Design PrivacyChecker

  • PrivacyChecker enumerates contextual information flows, judges each as permissible or not, and excludes or abstracts data marked No; optional guidelines customize final generation.Flows are represented using sender, recipient, subject, information type, and transmission principle.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Experiment

  • The primary action-based trajectory evaluation uses 493 PrivacyLens cases, with ConfAIde and CultureBank extensions; it measures leakage, helpfulness, and helpful-case adjusted leakage.The adjusted leak rate is calculated only for cases where the agent’s output is helpful, excluding nonresponses or irrelevant outputs.Source: §3.3 · pp. 4–5

3.3.2 Ablation Study

  • Removing judgment verbalization worsens leakage; ground-truth flows reduce it further, while optional privacy guidelines yield modest, consistent gains, especially for GPT-4o.The ablation evaluates GPT-4o and Qwen3-14B-NT on PrivacyLens.Source: §3.3.2 · p. 6, Table 2 · p. 6

4 Privacy under MCP and A2A Protocols

The section adapts PrivacyLens into live MCP and MCP+A2A workflows, then evaluates three ways to deploy PrivacyChecker. PrivacyChecker reduces leakage across static and live settings, while live and more complex workflows present additional challenges.

4 Privacy under MCP and A2A Protocols

  • PrivacyLens-Live uses MCP and A2A as testbeds, while PrivacyChecker is protocol-agnostic and designed to work with other agent systems.Its modular prompt-based reasoning and tool-call monitoring do not depend on MCP- or A2A-specific APIs or assumptions.Source: §4 · p. 6

4.1 Constructing Live MCP Benchmark

  • The live MCP benchmark implements Gmail and Notion tools, using 150 samples focused on those platforms and enabling retrieval in both tools and email sending in Gmail.The setup connects an LLM agent to MCP servers and tests each static sample in a contextual environment.Source: §4.1 · pp. 6–7

4.2 Constructing Live MCP + A2A Benchmark

  • The MCP + A2A benchmark models two-agent scenarios with a sender’s agent and a receiver’s agent communicating through A2A and using MCP tools.In the example workflow, the receiver’s agent retrieves information and replies through Gmail MCP.Source: §4.2 · p. 7

4.3 Deploying PrivacyChecker in MCP

  • PrivacyChecker can be deployed in system prompts, inside MCP tools, or as a standalone checking tool; each option trades off consistency, tool-specific flexibility, and monitoring complexity.A standalone checker reviews the draft and trajectory and requires approval before the sending tool is called; privacy-guidance integration in A2A remains future work.Source: §4.3 · pp. 7–8

4.4 Evaluating PrivacyChecker in Live MCP and MCP + A2A Benchmarks

  • Leakage is generally higher in live than static benchmarks; PrivacyChecker reduces leak rates across deployment strategies, with comparable results for MCP and MCP + A2A.Leak-rate differences among deployment strategies are relatively small, so deployment choice can depend on other trade-offs.Source: §4.4 · p. 8, Table 3 · p. 8
  • Noisy live trajectories with failed or redundant tool calls can yield partial retrieval and leak confidential information; providing all relevant content directly produced cleaner trajectories and more privacy-preserving outputs.The case study contrasts live MCP and MCP + A2A runs with a short, clean static trajectory; Figure 3 depicts the comparison and PrivacyChecker mitigation.Source: §4.4 · p. 8, Figure 3 · p. 9

4.5 Generalizing to More Complex Agent Workflows

  • Across 2-tool and 3-tool workflows, PrivacyChecker reduces leakage, but leakage increases in 3-tool cases and failures often involve missed or ambiguous information-flow extraction.The expanded benchmark adds Google Calendar, Slack, and Messenger and includes 36 new cases.Source: §4.5 · pp. 8–9, Table 4 · p. 9

5 Conclusion

PrivacyChecker applies contextual-integrity reasoning to LLM agents and substantially reduces leakage while preserving helpfulness. PrivacyLens-Live evaluates privacy in dynamic MCP and A2A workflows, where the paper reports higher leakage risks than in prior single-agent probes.

5 Conclusion

  • PrivacyChecker reduces leakage by over 75% across diverse models while preserving task helpfulness.Reported leak rates fall from 36.08% to 7.30% on DeepSeek-R1 and from 33.06% to 8.32% on GPT-4o.Source: §5 · p. 9
  • PrivacyLens-Live transforms static privacy benchmarks into dynamic MCP and A2A agent workflows that expose higher leakage risks than prior single-agent probes.Source: §5 · p. 9
  • The authors present PrivacyChecker as a modular, model-agnostic framework that prompts agents to identify and evaluate information flows at inference time.They aim for the released data and code to support further work on benchmarks and mitigation strategies.Source: §5 · p. 9

Limitations

The paper identifies evolving protocols, limited tool coverage, and residual reasoning and robustness issues as boundaries on its evaluation and mitigation approach.

Limitations

  • PrivacyLens-Live uses current MCP and A2A implementations and supports a limited number of tool integrations, so the benchmark needs updating as protocols evolve and broader tools and workflows are added.Source: Limitations · p. 10
  • Residual leakage can arise from reasoning errors and judgment-action mismatches, while memory poisoning or contextual ambiguity could disrupt flow extraction or privacy judgment.The authors identify stronger alignment, flow-tracking, and robustness as areas needing attention.Source: Limitations · p. 10

A Judge–Action Gap in PrivacyLens Evaluation

The appendix reports trajectory-level probing-question accuracy alongside action-based evaluation and provides details on the PrivacyChecker prompt and a supervisor comparison.

A Judge–Action Gap in PrivacyLens Evaluation

B PrivacyChecker Prompt

C Guideline Used in Ablation

The common privacy guideline used in the ablation study is provided in Table 11.

D Ground Truth Information Flow Used in Ablation

The passages identify a table containing the ablation study's ground-truth information flow and tables containing agent prompts for MCP and A2A scenarios.

D Ground Truth Information Flow Used in Ablation

E Action Agent Prompt

  • Tables 12, 14, and 13 show the privacy-gated MCP agent prompt and the A2A sender and recipient prompts, respectively.Source: E Action Agent Prompt · p. 12

F Comparison with CI-based supervisor

On the static Priva- cyLens dataset, PrivacyChecker achieves lower privacy leakage across models while maintaining comparable utility.

G Prompt using in building A2A Benchmarks

The passages describe a privacy-analysis prompt and identify tables for benchmark prompts, evaluation results, and ablation-study information flow.

  • Tables 15 and 16 contain the prompts for making agent cards and converting instructions, respectively.Source: G Prompt using in building A2A… · p. 12
  • Table 6 reports trajectory-level probing-question accuracy and action-based evaluation results from PrivacyLens with a privacy-enhanced prompt.Source: Table 6 · p. 13
  • Table 7 presents PrivacyLens and PrivacyChecker action-based evaluation results for the ConfAide dataset.The caption defines LR as leak rate, Help as helpfulness, and LRh as adjusted leak rate for cases with Help score above 2.Source: Table 7 · p. 13
  • Table 8 presents PrivacyLens and PrivacyChecker action-based evaluation results for the CultureBank dataset.The caption defines LR as leak rate, Help as helpfulness, and LRh as adjusted leak rate for cases with Help score above 2.Source: Table 8 · p. 13
  • Table 9 contains the ground-truth information flow used in the ablation study.Source: Table 9 · p. 13
  • The prompt instructs the agent to analyze privacy implications of information collected from the previous observation before answering.Source: G Prompt using in building A2A… · p. 14

Information Flow Analysis

The prompts represent communication as distinct information flows, then judge whether each flow may be shared. They also define agent context and convert recipient instructions into direct requests to sender agents.

Information Flow Analysis

  • Each flow records the information type, subject, sender, recipient, and sharing purpose; separate flows are required when information differs by time, place, people, purpose, or category.The prompt limits included details to those necessary, asks for an A/B acceptability judgment, and requires a high-level summary instead of quoting or paraphrasing data when the answer is B.Source: Information Flow Analysis · p. 14

Available Tools

  • The setup gives agents access to Gmail or Notion as needed and describes users and their relationships, keeping sender-only relationships private from the recipient’s agent.The scenario has separate sender and recipient agents, each operating with descriptions of the user it serves and relevant relationships.Source: Available Tools · pp. 16–17, Information Flow Analysis · p. 16

EXAMPLE INPUT

  • Instruction conversion rewrites the recipient’s request as a first-person direct command, preserving its intent and details while including the sender’s name and recipient’s email.It replaces the recipient’s name with “I” or “me” and omits the relationship between the parties.Source: EXAMPLE INPUT · p. 19

Related papers

Loading 2509.17488v1…