PaperlayerView exact revision on arXiv

Source-linked AI summary

The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration

Vaidehi Patil, Elias Stengel-Eskin, Mohit Bansal

arXiv:2509.14284v1cs.CRcs.AIcs.CL

TL;DR

When multiple agents share individually innocuous information, an adversary may combine it with auxiliary knowledge to infer sensitive attributes that no single agent reveals. The paper formalizes and evaluates this compositional leakage in balanced adversarial and benign scenarios, testing intent-based and collaborative defenses. ToM improves sensitive-query blocking but costs benign utility, while CoDef achieves a more favorable balance.

  • Problem

    Single-agent privacy approaches do not account for sensitive information inferred by composing innocuous outputs across agents, often with partial background knowledge.

    Source: §1 · p. 2

  • Method

    The paper models cross-agent leakage in controlled adversarial and benign scenarios, then evaluates ToM intent reasoning and CoDef's shared-context voting defense.

    Source: §1 · p. 2

  • Results

    ToM blocks 89-97% of sensitive queries across models, while CoDef blocks 86-90% and retains higher benign success, including 70.2% on Qwen, 69.7% on Gemini, and 66.3% on GPT-5.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Defender collaboration achieves a more favorable balance between preserving benign utility and blocking sensitive inferences than single-agent ToM reasoning.

    Source: §7 · p. 12

    The framework focuses on inference-time threats with black-box defenders and realistic auxiliary information, and abstracts agents as functions rather than modeling LLM-specific parameters.

    Source: §2 · p. 4

Abstract

from arXiv · show

As large language models (LLMs) become integral to multi-agent systems, new privacy risks emerge that extend beyond memorization, direct inference, or single-turn evaluations. In particular, seemingly innocuous responses, when composed across interactions, can cumulatively enable adversaries to recover sensitive information, a phenomenon we term compositional privacy leakage. We present the first systematic study of such compositional privacy leaks and possible mitigation methods in multi-agent LLM systems. First, we develop a framework that models how auxiliary knowledge and agent interactions jointly amplify privacy risks, even when each response is benign in isolation. Next, to mitigate this, we propose and evaluate two defense strategies: (1) Theory-of-Mind defense (ToM), where defender agents infer a questioner's intent by anticipating how their outputs may be exploited by adversaries, and (2) Collaborative Consensus Defense (CoDef), where responder agents collaborate with peers who vote based on a shared aggregated state to restrict sensitive information spread. Crucially, we balance our evaluation across compositions that expose sensitive information and compositions that yield benign inferences. Our experiments quantify how these defense strategies differ in balancing the privacy-utility trade-off. We find that while chain-of-thought alone offers limited protection to leakage (~39% sensitive blocking rate), our ToM defense substantially improves sensitive query blocking (up to 97%) but can reduce benign task success. CoDef achieves the best balance, yielding the highest Balanced Outcome (79.8%), highlighting the benefit of combining explicit reasoning with defender collaboration. Together, our results expose a new class of risks in collaborative LLM deployments and provide actionable insights for designing safeguards against compositional, context-driven privacy leakage.

1 INTRODUCTION

The introduction frames compositional privacy leakage as a risk created when individually innocuous information is combined across agents, and presents a framework and defenses for studying it. Experiments show that ToM improves sensitive-query blocking, while CoDef offers a better balance with benign utility.

1 INTRODUCTION

  • Compositional privacy leakage occurs when multiple agents' individually innocuous outputs combine to reveal sensitive information not disclosed by any one agent.The introduction distinguishes this inference-time, cross-agent threat from memorization and direct disclosure.Source: §1 · p. 2
  • The evaluation framework splits sensitive attributes across agents' disjoint tables and tests both adversarial and benign compositions, with no single agent able to infer the target alone.Scenarios include mappings such as user IDs to names, IDs to products, and products to health conditions; the pipeline simulates siloed data.Source: §1 · pp. 2–3
  • Baseline CoT retains 64–76% benign success but provides limited protection, with sensitive-query blocking as low as 31–39% and adversarial success reaching up to 70%.The evaluation covers 119 balanced scenarios involving sensitive and benign inferences.Source: §1 · p. 3
  • ToM raises sensitive blocking rates to 89-97% across models, while CoDef blocks 86-90% and retains higher benign success; GPT-5 shows the strongest balanced outcome overall at ∼77–78% trade-off scores.CoDef benign success is 70.2% on Qwen, 69.7% on Gemini, and 66.3% on GPT-5; ToM benign success is 52.9-61.6%, compared with 65-76% for lighter CoT baselines.Source: §1 · p. 3
  • ToM defenders reason about possible adversarial intent, whereas CoDef lets peer defenders share context and vote, with one block vote sufficient to deny a query.The introduction reports that ToM reduces leakage at a greater cost to benign performance, while CoDef better preserves benign utility.Source: §1 · pp. 2–3

2 THREAT MODEL

The threat model considers defenders holding separate, individually benign mappings while an adversary combines their responses with background knowledge to infer a sensitive attribute. It focuses on inference-time, black-box threats and abstracts agents from their underlying LLM parameters.

  • The global sensitive attribute is absent from every individual data view but can be inferred through structured probing.Each defender holds only a unique, non-overlapping subset of benign entity-attribute mappings.Source: §2 · p. 4
  • The adversary’s state combines initialization-time auxiliary knowledge with interaction history, and a function g uses that state to recover the sensitive attribute.Source: §2 · p. 4
  • Defenders are schema-aware but content-blind to one another, while the adversary queries individual agents through black-box access.Source: §2 · p. 4
  • Compositional leakage occurs when the adversary reconstructs the sensitive attribute with high accuracy without any single agent explicitly revealing it.Source: §2 · p. 4
  • The framework targets inference-time threats, assumes black-box defender access and realistic auxiliary information, and abstracts agents as functions over structured views rather than LLM-specific parameters.Source: §2 · p. 4

3 EVALUATING LEAKAGE: CONTROLLED MULTI-AGENT SETUP

The evaluation constructs scenarios where sensitive targets are derivable only by combining agents’ partial data, then tests leakage through a controlled multi-agent protocol. It models adversary–defender interaction under partial observability and measures leakage and defense outcomes.

3 EVALUATING LEAKAGE: CONTROLLED MULTI-AGENT SETUP

  • Scenarios make sensitive targets derivable through composition across agents, although no single agent directly exposes the target.The evaluation uses semantically rich multi-agent scenarios in which sensitive attributes are derivable but not directly observable to any one agent.Source: §3 · p. 4

3.1 CONTROLLED PLANNING-EXECUTION PROTOCOL

  • The protocol gives the adversary a ground-truth plan before execution, isolating execution and information-flow risks from planning errors.The adversary first receives a high-level plan, then queries agents and collects responses; holding the plan fixed approximates an upper bound under optimal high-level planning.Source: §3.1 · p. 4

3.2 EVALUATION COMPONENTS

  • The setup gives each scenario multiple defenders with local tables, an adversary with auxiliary knowledge, and a sensitive target pursued through sequential queries.The adversary’s success is measured by reaching an absorbing state representing successful leakage.Source: §3.2 · p. 5

3.3 LEAKAGE METRICS

  • Leakage Accuracy checks whether the adversary infers the target, while Plan Execution Success requires every intermediate retrieval step to be correct.A single failed step makes the whole plan unsuccessful, separating retrieval failures from failures in composition or reasoning over retrieved fragments.Source: §3.3 · p. 5

3.4 ADVERSARY-DEFENDER INTERACTION AS AN INTERACTIVE POMDP

  • Adversary–defender interaction is modeled as a partially observable Markov decision process, since each side acts on partial observations rather than the other’s hidden state.Defenders respond to incoming queries, and collaborative policies may consult peers before answering or blocking.Source: §3.4 · pp. 5–6

3.5 EXPERIMENT DETAILS

  • Evaluation metrics cover leakage, sensitive blocking, benign success, balanced outcome, and paired overall success, with Qwen3-32B used as judge.The experiments use Qwen3-32B as the adversary, vary defender models and policies, and record exchanges, blocked queries, and final inferences.Source: §3.5 · pp. 7–8

4 EXPERIMENTAL RESULTS

The experiments compare defense strategies across models on benign and sensitive scenarios, using aggregate outcomes to assess privacy–utility balance. CoT offers limited sensitive-query protection, while ToM and collaboration improve blocking with differing utility trade-offs.

Defense effectiveness across models

  • 79.8% Balanced Outcome for Gemini-2.5-pro CoDef exceeds Self-voting’s 76.9%, while Qwen3-32B Self-voting reaches 78.7% versus CoDef’s 78.4%.Table 2 reports benign and sensitive outcomes alongside aggregate metrics for each model and defense strategy.Source: Table 2 · p. 8, §4 · p. 9

Simple CoT baselines

  • CoT blocks only 31.1% of Qwen3-32B sensitive runs, 34.1% for Gemini-2.5-pro, and 70.5% for GPT-5.Benign success remains 63–69% across models, but sensitive inference is not reliably blocked, particularly in smaller models.Source: §4 · p. 8

Defense effectiveness across models

  • ToM substantially improves sensitive-query blocking but can reduce benign utility, especially for Gemini-2.5-pro; defense effectiveness also varies by model.The paper notes that calibration may need to account for model capacity and reasoning ability.Source: §4 · p. 9

5 ANALYSIS

The analysis examines reasoning depth, plan execution, and a multi-step example to show how defenses affect compositional inference. ToM and collaboration are associated with deeper reasoning and sharply reduced sensitive-plan execution while retaining benign execution.

Reasoning depth and leakage

  • Baseline defenses are dominated by Depth 0 responses, whereas ToM and CoDef show more Depth 2 and Depth 3+ reasoning.Depth 0 covers direct answers or refusals; higher levels capture context linkage, risk discussion, and multi-step inference.Source: §5 · p. 9, Figure 3 · p. 10

Plan execution success

  • Sensitive-plan execution drops drastically under ToM and Defender Collaboration, while benign execution remains relatively high across methods.A plan counts as successful only when all its steps are correctly executed.Source: §5 · pp. 9–10, Figure 4 · p. 10

Qualitative multi-step inference

  • In the example, ToM blocks sensitive steps 2–4, while CoDef allows step 2 but blocks steps 3–4, leaving only partial inferences.The baseline sequence combines flight, payment, personal, and approval details to identify Jane Smith.Source: §5 · p. 10, Figure 5 · p. 11

6 RELATED WORK

The related work situates compositional leakage alongside research on memorization, inference, dialogue privacy, and multi-agent systems. It distinguishes this paper’s focus on distributed information fragments and cross-agent accumulation from single-model disclosure settings.

Privacy risks in LLMs

  • The paper distinguishes compositional inference across agents from memorization and single-model inference-time privacy violations.Its focus is sensitive attributes revealed when seemingly innocuous outputs from different agents are aggregated.Source: §6 · p. 11

Task decomposition and attacks

  • Unlike sequential decomposition attacks on a single agent, this work studies information distributed across agents with separate contexts.The paper frames this setting as closer to fragmented information in realistic environments.Source: §6 · p. 12

Inference risks and user profiling

  • The paper separates its setting from user profiling and other attacks framed around a single model or user-written text.It focuses on leakage emerging in decentralized multi-agent deployments.Source: §6 · p. 12

Contextual privacy

  • The paper argues that locally reasonable contextual behavior may still permit privacy violations when inter-agent context is ignored.The cited concern is cross-agent accumulation enabling compositional inference.Source: §6 · p. 12

Sanitization and synthetic data

  • The paper contrasts cross-agent compositional leakage with sanitization and privacy guarantees aimed at centralized, single-model disclosure.It states that combinations of outputs can leak information even when individual agents are sanitized or privacy-protected.Source: §6 · p. 12

Dialogue privacy and threat models

  • The paper addresses a gap in privacy-focused dialogue datasets with synthetic multi-agent scenarios where partial auxiliary knowledge supports inference across interactions.The scenarios are designed so no single agent reveals sensitive data.Source: §6 · p. 12

7 CONCLUSION

The conclusion presents an interactive multi-agent framework for compositional privacy leakage and two defenses: ToM and CoDef. It emphasizes the privacy–utility balance of collaboration and calls attention to coordinated safeguards and evaluation.

Framework

  • The framework models adversaries and defenders as agents with evolving states and formalizes leakage as an interactive game.It targets strategic composition of distributed, seemingly benign data fragments to infer sensitive global attributes.Source: §7 · p. 12

Defenses

  • The proposed ToM and CoDef defenses model adversarial intent individually or deliberate jointly, with collaboration offering a more favorable privacy–utility balance.The conclusion contrasts collaborative defender deliberation with single-agent ToM reasoning.Source: §7 · p. 12

Implications

  • The paper frames its framework as a step toward systematic evaluation and further research on collective defenses, adaptive reasoning, and principled privacy guarantees.It states that multi-agent privacy risks cannot be fully addressed by isolated safeguards alone.Source: §7 · p. 13

A PROMPTS

The prompts specify how agents route and answer queries, how an adversary’s intent is assessed across a question sequence, and how defenders decide whether to share information. Separate evaluators assess plan execution and sensitive final inference, while a rubric classifies response reasoning depth.

Agent response prompt

  • If an agent cannot answer from its own context, it selects a likely knowledgeable agent and frames a context-appropriate query rather than forwarding the original verbatim.The prompt requires a respond-or-ask JSON action and requests chain-of-thought before the JSON.Source: A PROMPTS · p. 17

Adversarial-intent prompt

  • The adversarial planner evaluates the entire question sequence for access to all or most parts of a known sensitive combination and reports suspected intent and goal.Its task is to break a high-level adversarial goal into step-by-step factual queries.Source: A PROMPTS · pp. 16–18

Collaborative Consensus Defense prompt

  • Collaborative Consensus Defense agents use local context, adversary history, and defender feedback to choose whether to answer, block, or return a vague response.Each step is represented as a (who_to_ask, what_to_ask) pair.Source: A PROMPTS · pp. 16, 18

Evaluation prompts

  • Plan-execution evaluation checks whether required plan steps produced their intended outcomes, treating partial or failed attempts as failures even when the plan was followed.It evaluates execution rather than whether the agent reached the final inference.Source: A PROMPTS · pp. 18–19
  • Final-inference evaluation checks whether the trajectory clearly reaches the specified sensitive inference, while the depth rubric distinguishes direct answers, contextual explanations, sensitivity discussion, and multi-step reasoning.Depth 3+ includes nested reasoning, indirect inference, and cross-agent effects.Source: A PROMPTS · pp. 19–20

B PLAN GENERATION EXAMPLE

The example plan for company travel data gathers flight, payment, personal-detail, and approval information in sequence. It also expresses those queries as defender-and-query pairs describing the agent and information type.

B PLAN GENERATION EXAMPLE

  • The example plan first retrieves employee IDs linked to Honolulu flights, then finds payment methods for those employees, obtains names and departments, and confirms trip approval status.The sequence draws on travel notes, payment records, personal details, and travel-approval information.Source: B PLAN GENERATION EXAMPLE · p. 21
  • The first two queries become pairs asking “me” for an employee ID-to-flight mapping and Gorosti for an employee ID-to-payment-method-and-amount mapping.The examples identify the source context for each requested mapping.Source: B PLAN GENERATION EXAMPLE · p. 21
  • The remaining pairs ask Slyvie for employee ID-to-name-and-department information and Kerstin for employee ID-to-approval-status information.Each pair specifies who to ask and what information to request.Source: B PLAN GENERATION EXAMPLE · p. 21

Related papers

Loading 2509.14284v1…