PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems

Faouzi El Yagoubi, Godwin Badu-Marfo, Ranwa Al Mallah

arXiv:2602.11510v3cs.AI

TL;DR

Output-only privacy benchmarks do not capture sensitive data leaked through internal channels in coordinating multi-agent systems. AgentLeak addresses this gap with a seven-channel benchmark evaluated across diverse domains, models, and execution traces. The results show lower final-output leakage but substantially greater total exposure when internal coordination channels are included.

  • Problem

    Existing privacy evaluations focus on final outputs or model-level norms, leaving leakage through inter-agent coordination and shared memory insufficiently measured.

  • Method

    AgentLeak instruments seven privacy-relevant communication pathways and evaluates coordinator-worker systems across 1,000 scenarios, five models, four domains, and 4,979 execution traces.

  • Results

    68.9% total system exposure: multi-agent configurations reduced C1 final-output leakage to 27.2% from 43.2% in single-agent mode while internal channels remained highly leaky.

  • Takeaways & Limitations

    Privacy risk in the evaluated coordinator-worker setting is strongly shaped by internal coordination channels that standard output-only audits cannot observe.

  • Takeaways & Limitations

    The reported rates rely partly on an LLM judge with a 7.4% false negative rate and English-only calibration, so they are conservative lower bounds with limited language coverage.

Abstract

from arXiv · show

Multi-agent Large Language Model (LLM) systems create privacy risks that current output-only benchmarks cannot measure. When agents coordinate on tasks, sensitive data may pass through inter-agent messages, shared memory, and tool arguments, all pathways that final-output audits typically do not inspect. We introduce AgentLeak, a benchmark for evaluating internal-channel privacy leakage in multi-agent LLM systems. AgentLeak instruments seven privacy-relevant communication pathways and provides a large-scale empirical evaluation focused on final outputs, inter-agent messages, and shared memory. Across 1,000 scenarios spanning healthcare, finance, legal, and corporate domains, five production LLMs (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Mistral Large, and Llama 3.3 70B), and 4,979 validated execution traces, we find that multi-agent configurations reduce final-output leakage (C1: 27.2% vs 43.2% in single-agent mode) compared with single-agent baselines but introduce internal channels that raise total system exposure to 68.9% (aggregated across C1, C2, C5). Inter-agent messages (C2) leak at 68.8%, compared with 27.2% for final outputs (C1), meaning that output-only audits miss 41.7% of violations. Across all five models and four domains, the pattern C2 $\geq$ C1 holds consistently. These results suggest, within the evaluated coordinator-worker setting, that privacy risk in multi-agent systems is strongly shaped by architectural coordination channels rather than final-output behavior alone: it arises from internal channels that remain invisible to standard output-level defenses.

I. INTRODUCTION

AgentLeak addresses a gap in output-only privacy evaluation by measuring sensitive-data leakage across internal coordination channels in multi-agent systems. Its benchmark and experiments show that these channels can expose substantially more privacy risk than final outputs.

  • Motivation: Sensitive data can pass through inter-agent messages, shared memory, tools, and other coordination pathways while final outputs appear compliant.A healthcare example produced a clean appointment confirmation while a delegation message carried the patient’s complete medical record.
  • Research gap: Current privacy benchmarks primarily audit final outputs or model awareness of privacy norms, leaving internal coordination insufficiently evaluated.The paper contrasts output auditing with the need to assess privacy handling during complex, multi-step tasks.
  • Contribution: AgentLeak instruments seven communication channels across 1,000 scenarios in healthcare, finance, legal, and corporate domains.The benchmark uses coordinator-worker topologies and includes sensitive-data vaults, allowed disclosure sets, and ground-truth labels.
  • Results: 27.2% versus 43.2%: multi-agent configurations reduced final-output leakage relative to single-agent mode.The evaluation covered five production LLMs and 4,979 execution traces.
  • Results: 68.9% total system exposure: internal channels raised exposure despite lower final-output leakage.Inter-agent messages leaked at 68.8%, while shared memory leaked at 46.7%; output-only auditing therefore misses most violations.
  • Implications: Multi-agent frameworks emphasize coordination but generally lack default controls for sanitizing inter-agent messages or restricting memory access.The reviewed frameworks propagate task context or expose shared state without systematic internal-channel access controls.

C. DEFENSE MECHANISMS

The paper argues that existing defenses concentrate on system boundaries even though multi-agent architectures create multiple internal leakage pathways. AgentLeak is positioned as a benchmark that jointly evaluates multi-agent topology, internal channels, execution traces, and reproducible ground truth.

  • Existing defenses: Most existing privacy defenses operate at system boundaries and provide limited coverage of inter-agent messages and shared memory.Output filters, guardrails, and privacy-focused prompts primarily protect external interfaces.
  • Benchmark gap: AgentLeak is the only reviewed framework satisfying all four comparison criteria: multi-agent support, internal-channel coverage, trace-based detection, and reproducible ground truth.The comparison is reported across the reviewed benchmarks and defense tools.
  • Problem statement: Multi-agent systems expand the attack surface because each agent can expose data through peers, tools, and memory during autonomous execution.The paper notes that a single task may generate many messages, tool calls, and memory operations, each constituting a potential leak.
  • Leakage channels: Current frameworks lack mechanisms to monitor or restrict internal-channel communication, unlike the controls available for external channels.The taxonomy distinguishes external channels from internal channels used for inter-agent coordination.
  • Leakage channels: AgentLeak classifies seven leakage pathways, including final outputs, tool interactions, telemetry, artifacts, inter-agent messages, and memory state.C2 covers coordination messages and C5 covers persistent agent memory that can leak data across sessions.

C. PRIVACY LEAKAGE FORMALIZATION

AgentLeak formalizes privacy leakage as unnecessary disclosure of vault fields through seven channels, evaluated against task-specific allowed sets. Its threat model spans benign operation, manipulated inputs, and compromised system components, while the benchmark design provides a reusable evaluation harness.

  • C. PRIVACY LEAKAGE FORMALIZATION: Privacy is operationalized through contextual integrity by protecting a defined vault of sensitive fields from unnecessary disclosure.The allowed set contains the minimal fields genuinely required for a task, turning data minimization into an empirical criterion.
  • C. PRIVACY LEAKAGE FORMALIZATION: AgentLeak distinguishes internal exposure, effective leakage across a trust boundary, and real harm from unauthorized exploitation.The benchmark measures leakage as a proxy for harm potential rather than measuring actual exploitation directly.
  • C. PRIVACY LEAKAGE FORMALIZATION: Leakage is assessed per channel against a statically defined allowed set based on task necessity.A scheduling task may allow patient names and preferences while disallowing diagnoses, with channel-specific policies permitting different fields.
  • C. PRIVACY LEAKAGE FORMALIZATION: A leakage event occurs when an unallowed vault field has semantic similarity above the calibrated threshold in any channel’s content.The formalization defines the vault V, allowed set A, channel set C, similarity function, and threshold τ.
  • D. THREAT MODEL: The threat model ranges from benign misconfiguration and overly helpful behavior to manipulated external sources and compromised tools, dependencies, or agent roles.A0, A1, and A2 represent progressively stronger adversarial capabilities.
  • D. THREAT MODEL: The evaluation excludes administrative infrastructure access, attacks on LLM training pipelines, and user-interface exploits that bypass the agent architecture.These boundaries define the threat settings covered by the benchmark.
  • IV. AGENTLEAK BENCHMARK DESIGN: AgentLeak is designed as a reusable framework independent of any specific domain or agent implementation.Its design includes scenario structure, domain coverage, data generation, an evaluation harness, and a detection pipeline.

A. DESIGN PRINCIPLES

AgentLeak is designed as a reusable, framework-independent benchmark with reproducible traces, structured scenarios, and seven-channel instrumentation. Its scenario schema specifies sensitive data, permitted disclosures, tools, topology, threats, and ground-truth outcomes across four privacy-sensitive domains.

  • Reusability: AgentLeak uses reusable scenario templates that separate task specifications from domain content, supporting community contributions without breaking evaluation consistency.
  • Framework Independence: Framework-independent JSONL traces capture privacy-relevant events uniformly across LangChain, CrewAI, AutoGPT, MetaGPT, and custom stacks.
  • Reproducibility: Fixed seeds, versioned prompts, documented configurations, and cryptographic scenario hashes support reproducibility and verification.
  • Scenario Schema: Each scenario specifies the task, private vault, allowed disclosure set, tools, agent topology, attack level, and ground-truth leakage outcomes.The allowed set operationalizes data minimization by restricting exposure to fields legitimately required for the task.
  • Domain Coverage: The benchmark covers healthcare, finance, legal, and corporate scenarios, with 250 scenarios per domain.

D. DATA GENERATION AND VALIDATION

AgentLeak combines synthetic and curated real-world data with expert-reviewed privacy labels and complementary detection tiers. Its validation strategy spans explicit canaries, structured identifiers, and semantically equivalent disclosures across the benchmark’s attack and privacy research scope.

  • Data Generation: AgentLeak combines synthetic and curated real-world data to increase realism while maintaining ethical compliance.Real-world sources include pre-anonymized, research-licensed PII samples and clinical notes.
  • Data Generation: Three sensitive-value tiers test detection from verbatim markers to format-valid identifiers and contextually plausible natural-language facts.
  • Validation: Two domain experts independently assess whether vault fields are required, and fields rated 3 or higher by both annotators enter the allowed set.Inter-rater reliability is measured with Cohen’s κ.
  • Detection: The three-tier detector combines canary matching, structured-field audits, and LLM-as-Judge analysis to identify exact, formatted, and semantic disclosures.The tiers attribute each leak to the lowest-numbered detector that identifies it.
  • Attack Taxonomy: The benchmark provides a 32-class taxonomy and six attack families to systematize attack coverage, methodological comparison, and defense-gap identification.

B. ATTACK FAMILIES

AgentLeak organizes agentic privacy threats into six attack families spanning direct manipulation, environmental corruption, persistence, coordination, reasoning, and evasion. The evaluation measures leakage, attack success, utility, and current instrumentation boundaries across these threats.

  • Attack Families: The taxonomy groups attacks into six families based on their injection surfaces and target mechanisms.
  • Attack Families: Prompt and instruction attacks directly manipulate agent behavior, while indirect and tool-surface attacks corrupt what agents perceive.
  • Attack Families: Memory and persistence attacks exploit retained state, enabling extraction after the original task, whereas coordination attacks weaponize inter-agent trust.
  • Attack Families: Reasoning and chain-of-thought attacks target how agents think, while evasion and obfuscation attacks bypass pattern matching through recoverable encoding.
  • Evaluation Scope: F5 and F6 are included in aggregate metrics but are not separately tracked at trace level; per-family ASR is reported only for F1–F4.Future versions are intended to extend family-level attribution to all six families.
  • Evaluation Metrics: The evaluation compares single-agent baselines with multi-agent topologies using channel leakage, weighted severity, attack success, and task success metrics.ASR applies to adversarial A1/A2 scenarios, while TSR measures whether the agent objective is achieved.

B. DETECTION PIPELINE VALIDATION

The detection pipeline is calibrated and compared against simpler baselines using held-out data, robustness checks, and human review. The evaluation spans five models and 4,979 traces, while full-scale channel coverage remains limited to C1, C2, and C5.

  • Baseline Comparison: The full three-tier pipeline achieves F1 0.943, outperforming regex-only detection and LLM-as-Judge alone through combined precision and semantic coverage.Regex-only detection captures 8% of leaks, while Tier 3 alone has recall 0.926 and FPR 6.2%.
  • Robustness: Judge agreement ranges from Cohen’s κ = 0.82 to 0.87 across Qwen-2.5-7B, Llama-3.1-8B, and GPT-4o-mini.GPT-4o-mini achieves FPR/FNR of 4.1%/6.9% on the reported labeled subset.
  • Human Validation: Human review reports 94% accuracy with a 95% CI of [87.4%, 97.8%], with disagreements concentrated in ambiguous paraphrased medical terminology.
  • Experimental Setup: The benchmark evaluates five production LLMs across 1,000 scenarios and yields 4,979 validated traces after excluding 21 incomplete or timed-out traces.

E. RESULTS

Across paired traces, multi-agent systems lower final-output leakage but expand exposure through internal coordination channels. Inter-agent messages are the dominant vulnerability, and output-only audits miss substantial internal leakage.

  • Architecture comparison: 27.2% versus 43.2%: multi-agent configurations reduce C1 final-output leakage compared with single-agent execution.The paired design evaluates both executions on the same scenario and model.
  • Channel-level leakage: 68.8%: C2 inter-agent messages leak more often than C1 final outputs at 27.2%, while C5 shared memory leaks at 46.7%.Internal channels average 57.8% leakage, or 2.1× the external C1 rate.
  • Cross-model and domain consistency: C2 exceeds C1 for every tested model and all four domains, identifying inter-agent communication as the primary vulnerability.The model-level comparison includes GPT-4o, GPT-4o-mini, Mistral Large, Llama 3.3 70B, and Claude 3.5 Sonnet.
  • Audit gap: 41.7% of traces satisfy H1: C1 is safe while C2 or C5 leaks, so output-only audits miss these violations.The H1 rate measures a different subset from the 27.2% C1 leak rate and is not additive with it.
  • Defense effectiveness: Current defenses protect external channels more effectively than internal channels, with output sanitization showing zero impact on internal leakage.On the defense-evaluation subset, output sanitization reduced C1 leakage to 1% while leaving internal channels unaffected.

Finding 5: All Evaluated Frameworks Lack Internal Privacy Mechanisms

The evaluation finds that existing frameworks lack internal-channel privacy mechanisms while leakage extends across tools, logs, and semantic disclosures. Secondary channels can leak sensitive data even when user-facing outputs remain clean.

  • Framework coverage: AgentLeak is the only reviewed framework satisfying coverage, internal-channel interception, trace-based leakage measurement, and reproducible ground-truth criteria.The benchmark instruments multi-agent topologies and internal channels such as C2 and C5.
  • Framework-level vulnerability: 28–35%: all four evaluated frameworks show similar baseline C2 leakage, rising to 68.8% under adversarial scenarios.The tight baseline spread across structurally different frameworks supports a systemic vulnerability interpretation.
  • Attack success: 82.9%: multi-agent coordination attacks have the highest attack success rate, using inter-agent trust and shared memory for extraction.Prompt/instruction and tool-surface attacks also exceed 75%, at 79.8% and 78.9%.
  • Secondary channels: 62%–86%: C3 tool-input leakage occurs across all models, while C6 system-log leakage consistently exceeds tool invocation leakage.These channels were assessed on a representative sample of 100 scenarios per model rather than the full trace set.
  • Secondary channels: Over 65% of scenarios show C3 or C6 leakage while C1 remains clean, exposing a shadow leakage pathway beyond user-facing audits.These secondary channels bypass user-facing redaction and internal sanitization mechanisms.
  • Leakage type analysis: 82% of identified leaks are semantic, so pattern-matching defenses miss most violations and semantic analysis is needed for broader coverage.Tier 1 canary detection accounts for 8%, while Tier 2 structured-field detection accounts for 10%.
  • Model and domain analysis: Across all five models, C2 leakage exceeds C1 leakage, with Claude 3.5 Sonnet lowest on both channels but still leaking internally.Claude 3.5 Sonnet records C1=3.3% and C2=28.1%; GPT-4o shows a 4.5× internal-to-external gap.

F. SECURITY-UTILITY TRADEOFF ANALYSIS

The paper frames internal over-sharing as a privacy-risk and utility tradeoff rather than equating every internal disclosure with an actual breach. Strict controls reduce leakage but can impair legitimate coordination.

  • Security–utility tradeoff: Strict privacy settings reduce task success to 35–65% when they prevent any sensitive data sharing between agents.The paper therefore motivates selective disclosure of minimum-necessary fields through framework-level policies.
  • Violation, risk, and breach: The benchmark treats unnecessary internal disclosure as a policy violation and quantifies conditions that enable risk amplification, not every actual confidentiality breach.An actual breach requires sensitive data to reach an unauthorized party.
  • Violation, risk, and breach: 41.7% H1: output audits can pass while internal channels contain data that should not have been shared under strict minimization requirements.The benchmark adopts a conservative data-minimization perspective aligned with contextual privacy concerns.
  • Scope and limitations: The lower multi-agent C1 rate may partly reflect prompt-structure differences, so isolating coordination effects requires a controlled ablation.The limitation concerns interpretation of the 27.2% versus 43.2% C1 comparison.

B. IMPLICATIONS FOR SYSTEM DESIGNERS

The findings support full-channel auditing and privacy-aware deployment practices because internal channels can expose sensitive data even when output audits appear clean. The study also identifies scope and evaluation constraints that should shape interpretation and future deployment decisions.

  • Auditing and deployment: 41.7% of violations were missed by output-only audits across 4,979 traces, supporting full-channel auditing before deployment.The paper recommends comprehensive trace logging and privacy impact assessments that explicitly address internal communication channels.
  • Detection: Hybrid detection combines Presidio NER with an LLM judge to detect leakage under real task delegation and tool usage.The showcase was built with the AgentLeak SDK and CrewAI.
  • Architectural risk: Every tested model leaked PII through internal channels, and the failure was consistent across model families.The authors characterize the vulnerability as architectural rather than model-specific in the evaluated workflow.
  • Evaluation constraints: The evaluation covers four domains and 250 scenarios per domain but cannot capture every privacy-sensitive workflow.Unusual workflows, emerging applications, and other configurations may behave differently.
  • Evaluation constraints: The study tested only English scenarios and the coordinator-worker topology with two agents.Multilingual deployments and hierarchical, peer-to-peer, or larger topologies may exhibit different leakage profiles.
  • Evaluation constraints: 7.4% of Tier 3 detections are false negatives, so reported leakage rates are conservative lower bounds.The judge threshold τ = 0.72 was calibrated on English scenarios and may require retuning for other languages or specialized domains.
  • Evaluation constraints: The benchmark covers four major frameworks, but rapid ecosystem and framework changes require periodic reevaluation.Newer frameworks may have different vulnerability profiles.
  • Evaluation scope: AgentLeak’s large-scale analysis focuses on C1, C2, and C5, while C3 and C6 use targeted sampling and C4 and C7 appear in a case study.The design quantifies the main internal-channel mechanism without claiming complete quantitative evaluation of all seven channels.

E. FUTURE RESEARCH DIRECTIONS

The paper points toward framework-level privacy protections that minimize disclosure during coordination and extend monitoring beyond the evaluated channels and modalities. It also presents AgentLeak and its SDK as infrastructure for reproducible evaluation and future defense research.

  • Privacy-aware coordination protocols: Selective disclosure at the framework level is identified as the critical near-term priority for privacy-aware coordination.Open questions include formally verifiable data-flow policies and dynamic workflow management.
  • Privacy-preserving computation: Differential privacy and secure multi-party computation could enable coordination without exposing plaintext.Budget selection, composition, and performance overhead remain open challenges.
  • Multimodal and automated analysis: Extending analysis to vision, audio, and human-in-the-loop patterns could increase production coverage and proactive leakage detection.The paper proposes combining multimodal evaluation with static and dynamic data-flow analysis for continuous monitoring.
  • Adversarial robustness: 80% attack success for coordination attacks motivates defenses targeting inter-agent trust exploitation.The paper identifies this threat as absent in single-agent settings.
  • Benchmark infrastructure: AgentLeak instruments seven privacy-relevant channels, but the main quantitative analysis does not cover all seven equally.The benchmark and SDK are publicly available under an MIT license with scenarios, detection components, integrations, and reproducibility materials.
  • Mitigation directions: A prototype sanitization interceptor reduced internal leakage from 31.5% to 2.4%, with a 4.7-point drop in task success.The conclusion frames message sanitization, field-level memory controls, and default full-channel auditing as framework-level changes.
  • Benchmark infrastructure: The benchmark records event-level traces containing channel messages, model and scenario metadata, and leakage metrics.The appendix describes JSON and JSONL trace formats plus configuration and exclusion policies.
  • Detection limitations: False negatives include paraphrased medical facts that preserve clinical meaning without explicit diagnosis codes.False positives can include benign structured identifiers matching SSN patterns.

APPENDIX. SDK SHOWCASE IMPLEMENTATION DETAILS

The SDK showcase implements a three-agent portfolio-management workflow that processes sensitive financial data with real tool calls and hybrid leakage detection.

  • Application architecture: The showcase uses three CrewAI agents: Senior Research Analyst, Senior Financial Analyst, and Investment Advisor.Integration tests process IBANs, tax brackets, and trade history with real market-data APIs and internal calculators.
  • Vulnerability design: The application intentionally passes client PII in prompts, full financial profiles between agents, and IBAN verification requests to a calculator tool.These flaws expose the workflow’s sensitive-data pathways for evaluation.
  • Benchmark configuration: Five LLMs are evaluated on the same workflow using hybrid detection with Presidio NER and LLM-as-Judge semantics.The models are queried through OpenRouter with the benchmark configuration.
Loading 2602.11510v3…