PaperlayerView exact revision on arXiv

Source-linked AI summary

Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control

Minsun Shim, Ramisha Raida Karim, Ruthwik Jakkula, Kaiwen Zhou, Xin Liu, Xin Eric Wang, Zhou Li

arXiv:2609.14003v1cs.CRcs.AI

TL;DR

LLM-agent privacy defenses remain vulnerable when the model makes privacy decisions from adversary-controlled context. FLOWSEAL moves enforcement to a tool-level interceptor grounded in provenance and information-flow control; evaluations report low leakage while retaining task utility.

  • Problem

    Privacy enforcement based on an LLM's judgment over adversary-controlled context leaves the enforcement mechanism exposed to the same context the attacker manipulates.

    Source: §I · p. 2

  • Method

    FLOWSEAL combines provenance-derived ownership labels, a cooperative consent layer, and a mandatory tool-level interceptor using information-flow control.

    Source: §I · p. 2

  • Results

    FLOWSEAL limits item-level leakage to 0–3.3% while retaining 72.4% benign-task utility across evaluations of three benchmarks, five defense baselines, and eight attacks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results suggest reliable LLM-agent privacy protection can be achieved by relocating enforcement beyond adversarial framing rather than relying solely on the model's judgment.

    Source: §X · p. 14

    The remaining attack surface is errors in declassification, a separate LLM check of whether proposed output is semantically unrelated to protected records.

    Source: A. Information Flow Lattice · p. 6

Abstract

from arXiv · show

Personal AI agents built on large language models (LLMs) are increasingly given access to a user's private data and communications in order to provide personalized assistance. This access creates a persistent privacy risk: the agent must decide whether a given sensitive information should be disclosed to a particular party. Existing defenses address this by making the agent's backend LLM more privacy-preserving through stronger system prompts, training, or explicit consent-checking procedures, but this approach has a structural challenge: whenever enforcement is a judgment the LLM makes over the same conversational context an adversary controls, the enforcement mechanism and the attack surface coincide. We demonstrate this against existing defenses with three new attacks that require only ordinary agent interaction and no prompt injection: Collaborative Workspace Lure reframes an extraction attempt as collaborative work; Semantic Obfuscation Attack induces disclosure through omission rather than through anything the agent writes; and Channel Decoupling Attack splits the extraction request and the disclosure across independent channels. All three achieve substantially higher leak rates than the attacks these defenses were originally designed to withstand. Guided by this observation, we present FLOWSEAL, a defense that enforces confidentiality through a tool-level interceptor outside the LLM's context, grounded in data provenance and an information-flow-control lattice with controlled declassification. Evaluated across three benchmarks, five prompt-based baselines, and eight attacks, including a real agent executing live tool calls through MCP, FLOWSEAL reduces leak rates to near zero (e.g., 52.2% to 0.5% against Collaborative Workspace Lure) while preserving task utility, regardless of the underlying LLM backend.

I. INTRODUCTION

Personal agents use private records to assist users, but appropriate disclosure depends on the recipient and context. The paper identifies a structural weakness in defenses that leave privacy enforcement to an LLM interpreting adversary-controlled context.

I. INTRODUCTION

  • Existing defenses rely on the LLM to recognize privacy decisions, so adversarial framing can prevent the checks from being invoked.Source: §I · p. 1
  • Against SPR-D2, the new attacks achieve 43.8–75.6% item-level leakage, versus 1.9–3.0% for attacks discovered by SPR.Source: §I · p. 2
  • FLOWSEAL uses provenance labels, a mandatory tool-level interceptor, and information-flow control to mediate writes and shares outside the agent’s conversational context.Its release-boundary logic is deterministic, while semantic declassification remains a separate LLM call.Source: §I · p. 2
  • 0–3.3% item-level leakage with 72.4% benign-task utility; on live MCP, SOA leakage falls from 34.7% under PrivacyChecker to 0%.FLOWSEAL was evaluated against five defense baselines and eight attacks across three agent LLM backends.Source: §I · p. 2
  • CWL reframes extraction as collaborative work, SOA reveals information through omission, and CDA separates the extraction request from disclosure across channels.The attacks require ordinary agent interaction and do not use prompt injection.Source: §I · pp. 1–2

II. BACKGROUND

  • Background work covers agent tool workflows, benign-task oversharing, active extraction, and defenses tied to particular representations or semantic checks; FLOWSEAL follows fine-grained IFC.Source: §II · pp. 2–3

III. PROBLEM FORMULATION

The problem formulation models a personal assistant handling external parties’ requests and evaluates defenses by balancing disclosure against utility. It specifies the actors, adversary capabilities, and out-of-scope threats.

III. PROBLEM FORMULATION

  • The setting assumes a personal assistant handles conversations between its user and external parties.Source: §III · p. 3
  • The data subject owns the sensitive personal data, and the trusted agent has legitimate access to it and handles other parties’ requests.Source: §III · p. 3
  • The external recipient is the adversary, seeking sensitive information without authorization and lacking direct access to the data, system prompt, or memory.Source: §III · p. 3
  • The adversary may adapt multi-turn requests to manipulate the agent’s judgment, while the agent and its tools are assumed trustworthy.Source: §III · p. 3
  • Leakage includes direct or indirect disclosures, such as paraphrases, inferences, confirmations, and selective edits or other revealing actions.Source: §III · p. 3
  • The threat model excludes compromised tools, poisoned tool metadata, compromised infrastructure, and traffic-fingerprinting side channels.Prompt injection is also outside the paper’s defense focus.Source: §III · p. 4
  • A defense must limit disclosure while meeting a minimum utility level; blocking every output would fail that constraint.Disclosure includes direct, indirect, and derived forms, while utility measures task completion quality.Source: §III · p. 4
  • The defense objective formalizes disclosure and task utility, with γ specifying the minimum acceptable utility.Source: §III · p. 4

A. Review of SPR Defense

SPR’s D2 defense uses a state machine for consent and identity checks, but its natural-language triggers can be bypassed by reframing or splitting disclosure tasks. Three new attacks exploit these weaknesses and achieve higher leak rates than SPR’s attacks.

A. Review of SPR Defense

  • SPR iteratively searches for attacks and strengthens defenses through simulated interactions and optimizer feedback.The examples use a scenario in which Emily attempts to obtain Mark’s mental-health diagnosis from Jane’s agent.Source: A. Review of SPR Defense · p. 4
  • SPR’s A1 fabricates consent and urgency, while A2 impersonates the data subject and then cites the fake consent.Source: A. Review of SPR Defense · p. 4
  • SPR-D2 adds consent and identity checks in a state machine, and its authors found comprehensive direct prompts substantially more vulnerable.Figure 3 shows the D2 defense prompt discovered through iterative search.Source: A. Review of SPR Defense · p. 4, Figure 3 · p. 4

B. Three New Attacks

  • D2 activates only when the LLM classifies an interaction as an information request, and its natural-language state guards lack system-level anchors.Reframing disclosure can therefore bypass the state machine even when its downstream logic is rigorous.Source: B. Three New Attacks · p. 4
  • CWL frames extraction as collaborative editing, leading the agent to write sensitive information into a shared workspace without triggering consent checks.Source: B. Three New Attacks · p. 5
  • SOA reveals which records are genuine through the agent’s selective deletions, although it assumes the attacker knows candidate records.Source: B. Three New Attacks · p. 5
  • CDA separates the request in email from disclosure on Notion, so D2 cannot connect the channels or block the write.Source: B. Three New Attacks · p. 5
  • All three new attacks achieve higher leak rates against D2 than the attacks discovered by SPR.Source: B. Three New Attacks · p. 5

V. DEFENSE: FLOWSEAL

FLOWSEAL moves privacy enforcement out of the conversational context controlled by an attacker and into a mandatory tool-call layer. Provenance labels, two-layer mediation, and information-flow control govern when derived content may be released.

V. DEFENSE: FLOWSEAL

  • FLOWSEAL combines provenance-based ownership labels, a mandatory tool-call interceptor, a cooperative prompt-and-API layer, and an information-flow-control policy with controlled declassification.Release decisions consult ownership metadata rather than asking the LLM to infer sensitivity.Source: §V · p. 5

A. Information Flow Lattice

  • The three-level lattice labels sensitive records ⊤, the agent’s context agent after a protected read, and external sinks ⊥.The lattice is described as necessary and sufficient for the threat model.Source: A. Information Flow Lattice · pp. 5–6
  • A write is permitted only when its content label is no higher than the destination’s, or controlled declassification deems it SAFE.Declassification returns SAFE only when the content does not derive from a ⊤-labeled record.Source: A. Information Flow Lattice · p. 6
  • FLOWSEAL’s confidentiality theorem rules out writing content derived from ⊤-labeled records to a ⊥ destination unless declassification erroneously returns SAFE.The proof relies on intercepted writes, taint propagation, and the content checker; checker errors remain the residual attack surface.Source: A. Information Flow Lattice · p. 6
  • Prior IFC approaches discussed here focus on execution integrity or indirect prompt injection, rather than leakage queries that leave execution integrity intact.The paper contrasts FLOWSEAL with label propagation that does not enforce privacy checks and with FIDES, whose policy enforcement is performed by the LLM backend.Source: A. Information Flow Lattice · pp. 6–7

B. Architecture

FLOWSEAL pairs cooperative privacy APIs with mandatory code mediation at the tool-call boundary. The interceptor tracks taint, checks destinations and content, and can guide the agent toward an authorized sharing path.

B. Architecture

  • A mandatory interceptor mediates every tool call; reads update taint, while writes and shares pass destination authorization and, when needed, an isolated content checker.The cooperative layer assists the agent but is not security-critical because the interceptor still blocks leakage if the agent skips its APIs.Source: B. Architecture · p. 7
  • The taint tracker records newly encountered protected records and gives derived content the joined authorization labels of its source records.A summary drawing on multiple records therefore requires authorization for each source record before release.Source: B. Architecture · pp. 7–8
  • At the release boundary, the interceptor extracts destination and content, applies the label policy, and invokes the declassification checker when needed.The isolated checker receives relevant tainted records and proposed output, then returns SAFE or BLOCKED; it is triggered for outbound writes and shares.Source: B. Architecture · p. 8
  • When a call is blocked or denied, the interceptor adds actionable guidance directing the agent to an appropriate privacy API rather than retrying the operation.Source: B. Architecture · p. 8
  • Four cooperative APIs let the agent discover protected records, check data owners, request consent, and submit proposed output for declassification.REQUESTACCESS can return approved rendered content or DENIED; DECLASSIFY can return SAFE or REDACTED content.Source: B. Architecture · pp. 7–8

A. Evaluation Setup

The evaluation combines three simulation benchmarks, five prompt-based defenses, and tests across multiple LLM backends. FLOWSEAL is implemented as an extension to the SPR tool-call path, with a separate live MCP setup described for real services.

1) Benchmarks and Attack Seeds:

  • The three simulation benchmarks contribute 105 SPR seeds, 378 PrivacyLens seeds, and 571 ConVerse seeds, adapted to SPR’s multi-turn format.SPR simulates tool-using agents, PrivacyLens emulates tools as text references, and ConVerse evaluates agent-to-agent extraction without tool calls.Source: 1) Benchmarks and Attack Seeds · pp. 8–9

2) Defense Baselines:

  • Five prompt-based defenses are compared under SPR: SPR-D1/D2/D3, PrivacyLens-Defense, and FAN-LCF with its input firewall disabled and output firewall retained.D1 adds consent verification, D2 embeds a two-state machine, and D3 follows an additional defense-search round from D2.Source: 2) Defense Baselines · p. 9

3) Models and Infrastructure:

  • DeepSeek-V3.2 is the default agent LLM, with GPT-4.1-mini and Gemini-Flash-2.5 used for backend evaluation; FLOWSEAL is implemented as a tool-call-path extension to SPR.DeepSeek-R1 serves as the default judge, and the tool-call extension requires no modification to tool implementations.Source: 3) Models and Infrastructure · p. 9

4) Metrics:

The section defines leakage and utility measures, then shows that the new attacks substantially weaken prompt-based defenses while FLOWSEAL brings leakage close to zero. FLOWSEAL’s results are reported alongside attack patterns that tend toward comprehensive, early disclosure once defenses are bypassed.

4) Metrics:

  • LRI measures the fraction of sensitive items leaked, LRB the fraction of runs with any leak, and LV how early multi-turn leakage occurs.LV does not apply to single-turn scenarios; utility rate is the proportion of evaluated responses that fulfill benign-task objectives without requesting sensitive information.Source: 4) Metrics · p. 9, 5) Utility · p. 9

B. Results of the Proposed Attacks

  • Under SPR defenses, proposed CWL, SOA, and CDA attacks raise LRI from D1 to D2 by 31.8% to 52.2%, 54.7% to 75.6%, and 13.3% to 43.8%, respectively.Even under D3, LRI shows no prominent decrease, remaining at 51.8% for CWL, 69.7% for SOA, and 39.7% for CDA.Source: B. Results of the Proposed Attacks · p. 10
  • SOA has the highest leak rate against every SPR defense variant, while CDA is lower than CWL because it requires two coordinated steps.SOA reveals information through omission, and the leakage metrics indicate that the new attacks tend toward comprehensive and early disclosure once a defense is bypassed, unlike the partial leakage of baseline attacks.Source: B. Results of the Proposed Attacks · p. 10
  • PrivacyLens-Defense reaches 89.9% LRI under CWL, and prompt-based defenses are insufficient against attacks combining multiple social-engineering tactics.The authors also report that LRB tracks LRI for their attacks, whereas SPR-A2/A3 show a larger LRI–LRB gap, suggesting more partial leakage in baseline attacks.Source: B. Results of the Proposed Attacks · p. 10

C. Results of FLOWSEAL

  • FLOWSEAL reduces LRI from 52.2% to 0.5% for CWL, from 75.6% to 2.3% for SOA, and from 38.5% to 2.2% for CDA.The authors attribute this protection to two-layer information-flow control; CDA retains slightly higher residual leakage because its split-channel writes are harder to link to protected records.Source: C. Results of FLOWSEAL · p. 10

D. Results of Benign Seeds

FLOWSEAL retains utility close to the strongest baseline while its component ablations show the role of the interceptor, content checker, and prompt. Additional tests examine tool choice, consent behavior, and backend-model changes.

D. Results of Benign Seeds

  • FLOWSEAL achieves 72.4% utility versus SPR-D1’s 75.9%, while substantially reducing leakage under the new attacks.FAN-LCF reaches 25.1% utility; the authors suggest output-only checking without provenance tracing leads to more aggressive blocking.Source: D. Results of Benign Seeds · p. 10, Table IV · p. 11

VII. ABLATION STUDY

  • With Google Slides and Slack replacing Notion for CWL scenarios, leakage ranges from 0.0–1.3% LRI across the three channels because every tool is intercepted.The tests substitute each tool’s equivalent shared-artifact action for page editing.Source: §VII · pp. 10–11
  • Disabling the interceptor raises CWL leakage from 0.5% to 28.4% LRI; fail-closed checking yields 1.3% LRI but zero utility, while fail-open yields 24.9% LRI and 79.4% utility.Removing the prompt raises leakage from 0.5% to 8.8% LRI while utility changes from 72.4% to 74.4%.Source: §VII · p. 11, Table V · p. 12
  • When consent is granted for legitimate requests, FLOWSEAL has lower authorized-disclosure rates and fewer consent requests, while successful disclosures take a similar number of turns.Baseline defenses have high ADRI/ADRB, Nconsent near 1, and Nturns of 5–6; Nturns counts only runs that successfully reach authorized disclosure.Source: §VII · p. 12, Table VI · p. 12
  • Under CWL, SPR-D2’s LRI is 52.2, 59.2, and 61.9 for DeepSeek-V3.2, GPT4.1-mini, and Gemini-Flash-2.5; FLOWSEAL’s is 0.5, 1.9, and 0.6, respectively.Changing the judge model has little reported impact: SPR-D2 LRI changes from 52.2 to 50.5 and 50.7, while FLOWSEAL changes from 0.5 to 1.0 and 0.8.Source: §VII · p. 12, Figure 9 · p. 12
  • FLOWSEAL’s cooperative prompt adds $0.017 per run over no defense, while blocked-write retries add calls and runtime as the agent reformulates or abandons a write.The prompt-only SPR-D2 setup has lower overhead than the no-defense setup because its responses are shorter.Source: §VII · p. 12

VIII. REAL-WORLD TESTS

In live Gmail and Notion tests through MCP, FLOWSEAL eliminates measured leakage on SOA, with modest added cost. The discussion notes remaining checker errors and unexhausted attack variants, while the conclusion emphasizes enforcement beyond adversarial framing.

VIII. REAL-WORLD TESTS

  • On live Gmail and Notion endpoints, FLOWSEAL reduces SOA LRI, LRB, and LV from 34.7%, 48.0%, and 34.7% under PrivacyChecker to 0.0% each.FLOWSEAL costs $0.052 ± 0.007 per run versus $0.042 ± 0.006 for PrivacyChecker; runs seed real accounts and the study is small-scale rather than a replacement for the larger SPR evaluation.Source: §VIII · p. 13

IX. DISCUSSION

  • The attack surface is not exhausted: split-and-aggregate attacks could divide a protected fact across individually innocuous writes that pass content checks.FLOWSEAL’s checker can miss paraphrases or compound sentences and can over-block benign writes that name a protected record without revealing its content.Source: §IX · p. 13
  • FLOWSEAL’s tool-call enforcement can be complemented by OS-level monitoring and isolation, extending policy toward operations performed directly by tool backends.The paper describes eBPF probes, Linux Security Modules, and mobile permission boundaries as possible layers in a broader enforcement stack.Source: §IX · p. 13

X. CONCLUSION

  • The conclusion presents FLOWSEAL as provenance- and information-flow-control-based enforcement that reduces leakage across benchmarks, defenses, attacks, and a live MCP-connected agent.The authors frame enforcement beyond adversarial framing as a route to reliable privacy protection and report making the implementation publicly available.Source: §X · pp. 13–14

APPENDIX A ETHICAL CONSIDERATIONS

The appendices describe ethical safeguards, related work, the tool-call interceptor, and walkthroughs showing it blocking disclosure while allowing unrelated assistance.

APPENDIX A ETHICAL CONSIDERATIONS

  • Test accounts contained only fictional, fabricated data, which was deleted after each run; the study also limited live-agent runs and avoided bulk seeding.The authors state these steps addressed privacy concerns and burden on service providers.Source: Appendix A · p. 15

APPENDIX B OTHER RELATED WORK

  • Related work spans instruction and execution defenses, while information-flow control labels data and restricts flows under a security policy.The appendix notes FLOWSEAL uses lightweight tool-call-level taint tracking tailored to agent workflows.Source: Appendix B · pp. 15–16

APPENDIX C TOOL-CALL INTERCEPTOR ALGORITHM IN FLOWSEAL

  • The interceptor executes reads and updates taint, while writes and shares are checked against destination trust and unauthorized tainted records before execution or blocking.Its algorithm also handles privacy API calls, including consent requests and record listings.Source: 1) Adversarial: CWL Attack Blocked · p. 16

APPENDIX D DEFENSE WALKTHROUGH

  • In the collaborative-workspace walkthrough, FLOWSEAL blocks a Notion write containing Mark’s diagnoses, even though the agent never requests access.The records are tainted, Emily is an unknown recipient, and a content check blocks the proposed disclosure; a later non-sensitive rewrite is accepted.Source: 1) Adversarial: CWL Attack Blocked · p. 16
  • FLOWSEAL allows a reply with general wellness resources because it is unrelated to Mark’s diagnosis records.The agent sends community-support, co-parenting, and self-care suggestions without requesting access to those records.Source: 2) Benign: Legitimate Request Released · pp. 16–17

APPENDIX E BENCHMARK DETAILS

The appendices detail the benchmark setups, describe how SPR’s A3 attacks and D3 defense evolve, and define the leakage measures used in evaluation.

APPENDIX E BENCHMARK DETAILS

  • The SPR setup adds Google Docs, Google Slides, and Canvas to its seven-app codebase and uses 105 seed scenarios across three attack variants, each run three times.PrivacyLens evaluation uses 378 non-overlapping accepted seeds; ConVerse evaluation uses 571 runnable configurations, each run once.Source: Appendix E · p. 17
  • ConVerse tests contextual extraction through plausible cooperative dialogue across travel planning, real estate, and insurance, with no simulated tool calls.The evaluation adapts its attacker prompts to SPR’s three-agent format and uses ConVerse’s original privacy judge.Source: Appendix E · p. 17

APPENDIX F EVOLUTION OF SPR ATTACK AND DEFENSE (A3/D3)

  • SPR’s A3 moves beyond forged consent, using contextual cues or plausible justifications to elicit disclosure without the consent step; D3 adds explicit refusal beyond consent authentication.Figure 10 presents the D3 prompt, which requires direct verification from the data subject.Source: Appendix F · p. 17, Figure 10 · p. 18

APPENDIX G METRIC DEFINITION

  • Item-level and binary-level leak rates measure, respectively, the fraction of sensitive items leaked and the fraction of runs with at least one leak.Both use configuration-mean aggregation, first averaging within scenarios and then across scenarios so each scenario contributes equally.Source: Appendix G · pp. 17–18
  • Leak velocity scores when an item is first leaked among monitored sender actions, counting only its first disclosure and aggregating per-run scores by configuration mean.A value of 1 means all sensitive items leak at the first monitored action; 0 means no sensitive information leaks.Source: Appendix G · p. 18

APPENDIX H EXAMPLE TRACES CASE STUDY

The example traces contrast FLOWSEAL’s tool-boundary enforcement with prompt-based or advisory judgments, showing both blocked disclosures and a registration-gap failure.

A. SPR-D2 vs FLOWSEAL on same task

  • On the same CWL-Notion task, D2 is bypassed by one reclassification step, while FLOWSEAL’s outbound write check blocks the leak despite a mistaken safe judgment by its declassification layer.Table VIII presents the two defenses’ traces for this task.Source: Table VIII · p. 18, A. SPR-D2 vs FLOWSEAL on same task · p. 18

B. FLOWSEAL’s Successful and Failure defense against ConVerse attack

  • The ConVerse traces show both a successful block of a reframed extraction attempt and a failure that releases an unregistered health detail.Figure 11 marks the attacker turn, agent reasoning and draft, interceptor decision, and outcome; red highlights identify protected content at risk.Source: B. FLOWSEAL’s Successful and Failure… · p. 18, Figure 11 · p. 18

C. PrivacyChecker Traces

  • PrivacyChecker allows the live SOA draft because attacker-supplied page language labels it treatment coordination, and the diagnoses are written verbatim to Notion.The agent’s draft follows a cleanup request involving mixed true and false candidate entries about Mark’s mental health.Source: C. PrivacyChecker Traces · pp. 18–19
  • FLOWSEAL blocks the same draft at the MCP write while consent is pending, and its decision is independent of the attacker’s HIPAA headers and cleanup framing.Subsequent rewrites that retain diagnostic content are also blocked, so no sensitive diagnosis facts reach Notion.Source: C. PrivacyChecker Traces · p. 19
Loading 2609.14003v1…