PaperlayerView exact revision on arXiv

Source-linked AI summary

ContextLeak: Exfiltrating LLM Agent Context via Malicious Tools

Yuqi Jia, Ruiqi Wang, Patrick Li, Yuepeng Hu, Peinian Li, Neil Gong

arXiv:2608.27800v1cs.CRcs.AI

TL;DR

ContextLeak addresses the gap that malicious-tool attacks generally do not induce agents to pass runtime context as input arguments, a necessary step for exfiltration. It crafts tool names and descriptions with an attack LLM fine-tuned by reinforcement learning on diverse shadow contexts, and evaluation finds strong transfer across victim contexts while existing defenses remain insufficient.

  • Problem

    Existing malicious-tool attacks focus mainly on tool selection or malicious transmission and largely leave inducing disclosure of runtime context as input arguments unexplored, despite its importance for exfiltration.

  • Method

    ContextLeak uses reinforcement learning to fine-tune an attack LLM that generates malicious tool names and descriptions inducing both tool selection and runtime-context disclosure.

  • Results

    ContextLeak is highly effective across substantially different victim-context distributions and backend LLMs, substantially outperforms adapted baselines, and defeats existing prevention- and detection-based defenses.

  • Takeaways & Limitations

    ContextLeak demonstrates that malicious tool metadata can induce runtime-context disclosure, motivating new defenses beyond existing selection and detection approaches.

  • Takeaways & Limitations

    Extending ContextLeak to execute after benign tools, thereby exfiltrating sensitive information embedded in execution trajectories, remains future work.

Abstract

from arXiv · show

Exfiltrating an LLM agent's runtime context -- such as the user prompt, execution trajectory, and tool list -- poses severe security and privacy risks to users. Such attacks can be carried out via malicious tools and typically require three conditions: (1) the agent selects the malicious tool for task execution, (2) the agent passes its runtime context as input arguments to the tool, and (3) the tool's implementation transmits these inputs to an attacker-controlled endpoint. Existing work primarily focuses on conditions (1) and (3), leaving condition (2) largely unexplored, despite its critical role in enabling successful context exfiltration. In this work, we bridge this gap by developing ContextLeak, a malicious tool attack that induces the agent to both select the tool and disclose its context as input arguments. We realize this attack by carefully crafting the tool's name and description using reinforcement learning. Specifically, ContextLeak employs an LLM, referred to as the attack LLM, to automatically generate the malicious tool's name and description. To improve attack effectiveness, we fine-tune the attack LLM via reinforcement learning on a set of shadow users with diverse, simulated agent contexts. Our key technical contribution is the design of novel reward functions tailored to the context exfiltration objective, enabling effective reinforcement-learning-based fine-tuning of the attack LLM. Extensive evaluation demonstrates that our attack remains highly effective even when the shadow users' contexts differ substantially from those of the victim users. Moreover, ContextLeak significantly outperforms existing malicious tool attacks when adapted to this setting.

1 Introduction

ContextLeak addresses the overlooked challenge of inducing an LLM agent to pass runtime context to a malicious tool, enabling exfiltration. It uses reinforcement learning to generate effective tool metadata and remains effective across differing victim contexts and existing defenses.

  • Motivation and Contribution: Runtime context may include prompts, conversation history, and installed tools, potentially exposing sensitive personal, financial, professional, or organizational information.
  • Motivation and Contribution: ContextLeak induces an agent to select a malicious tool and pass its runtime context as input arguments for exfiltration.The attack targets the underexplored second condition of context exfiltration, alongside tool selection and malicious transmission.
  • Method: The attack LLM generates human-readable malicious tool names and descriptions, then reinforcement learning fine-tunes it for stronger exfiltration performance.The method uses a context-exfiltration reward and strategies that guide candidate generation during fine-tuning.
  • Evaluation: ContextLeak substantially outperforms adapted prompt-injection, jailbreak, and malicious-tool baselines across user prompts, conversation histories, and tool lists.
  • Evaluation: ContextLeak remains effective across substantially different victim-context distributions and backend LLMs, while existing defenses are insufficient.

2 Related Work

Related work addresses malicious-tool selection, implementation, and detection, but generally cannot reliably induce disclosure of task-specific runtime context. ContextLeak targets this missing input-argument step, while existing defenses face evasion or utility trade-offs.

  • Runtime Context and Tool Use: An agent’s runtime context combines task-specific information, such as prompts and conversation history, with cross-task information, such as memories, retrieved records, and tool lists.
  • Runtime Context and Tool Use: Tool selection determines whether to invoke a tool, which tool to choose, and what data to provide as input arguments.
  • Existing Attacks: Prior attacks increase malicious-tool selection or embed malicious behavior in code, but typically do not control the data supplied to selected tools.
  • Existing Attacks: Because tools usually lack direct access to runtime context, existing attacks cannot reliably exfiltrate task-specific information without explicit argument disclosure.
  • ContextLeak: ContextLeak crafts tool metadata to induce both malicious-tool selection and passage of runtime context as input arguments.
  • Defenses: Text-based detectors can be evaded because ContextLeak does not depend on overt malicious instructions or suspicious semantic cues.
  • Defenses: Selection-robustness defenses often degrade benign tool-selection utility, creating a security–utility trade-off.
  • Defenses: File-access controls can mitigate attacks that read stored cross-task context but do not prevent ContextLeak from exfiltrating context supplied during tool invocation.

3 Problem Formulation and Threat Model

The paper formulates context exfiltration as inducing a malicious tool to be selected and receive a representation of the agent’s runtime context. Its threat model assumes black-box interaction, limited agent knowledge, and transfer across diverse victim contexts.

  • 3.1 Context Exfiltration Problem: The runtime context C includes installed-tool names and descriptions, including a malicious tool M.
  • 3.1 Context Exfiltration Problem: The attacker designs M so the agent selects it and provides a representation of C as input arguments, after which M transmits those arguments to attacker-controlled endpoints.
  • 3.1 Context Exfiltration Problem: This work focuses on crafting M’s name and description, while its malicious implementation can use existing techniques.
  • 3.2 Threat Model: The attacker publishes M on a tool platform and seeks to exfiltrate runtime context containing prompts, conversation history, and installed tools.
  • 3.2 Threat Model: The crafted metadata is intended to succeed across diverse prompts and sets of benign tools, regardless of the victim’s specific context.
  • 3.2 Threat Model: The attacker knows the target agent and supported tool protocols but need not know the victim’s backend LLM.
  • 3.2 Threat Model: The attacker lacks direct access to the victim’s prompt, conversation history, and benign-tool set.
  • 3.2 Threat Model: The attacker can deploy the target agent and interact with it as a regular user in a black-box manner.

4 ContextLeak

ContextLeak fine-tunes an attack LLM to generate malicious tool names and descriptions that induce both tool selection and runtime-context disclosure. It uses context-specific rewards and strategies extracted from prior candidates to improve reinforcement-learning-based generation.

  • Approach: ContextLeak uses reinforcement learning to fine-tune an attack LLM for generating malicious tool names and descriptions that satisfy both selection and argument goals.The attack LLM is guided by a specialized system prompt, while direct prompting alone produces suboptimal outputs.
  • Training Pipeline: Each fine-tuning iteration evaluates multiple candidate tools by appending them to shadow contexts and observing the target agent’s selection and input arguments.Candidates receive higher scores when selected and when their arguments closely match the corresponding shadow context.
  • Reward Design: Argument quality is evaluated by word-level edit similarity and length consistency between the shadow context C′ and generated arguments A.The length term penalizes large discrepancies, encouraging complete but concise reconstruction without extraneous content.
  • Reward Design: The reward function explicitly combines a binary selection reward with an argument reward for context reconstruction.Successful selection receives reward 1, while Rarg promotes accurate and complete exfiltration; λ balances context fidelity and length consistency.
  • Candidate Guidance: Past candidate outcomes are abstracted into reusable strategies that guide later generations of malicious tool names and descriptions.The strategy representation includes a name, definition, and illustrative example, such as “Persuasive Tool Framing.”

5 Evaluation

Evaluation shows that ContextLeak jointly induces malicious-tool selection and context passing, outperforming baselines across context types and transferring across backend models and user shifts. Its effectiveness is robust to prompt-length variation, while domain shifts introduce greater degradation than dataset shifts.

  • Experimental Setup: ContextLeak uses Qwen-3-8B as the attack LLM and compares against prompt-injection, jailbreak, and malicious-tool baselines adapted for context exfiltration.The evaluated backend models include four open-weight and three closed-source LLMs.
  • Ablation Studies: ContextLeak consistently outperforms ContextLeak-w/o-S, showing that strategy-guided candidate generation further improves reinforcement-learning fine-tuning.Without strategy guidance, ContextLeak-w/o-S already reaches MTSR 0.88 on user prompts and 0.80 on conversation histories, with EDS and ES close to 1.0.
  • Main Results: Existing baselines remain weaker because selection-oriented attacks do not explicitly optimize for passing context as input arguments.AMA and ToolTweak achieve only modest selection rates and poor context-passing performance, while judge-based refinement provides less informative signals when selection and argument passing fail separately.
  • Main Results: JudgeDeceiver, ObliInjection, and ToolHijacker perform strongly on tool selection but remain inferior to ContextLeak on context passing.For tool-list exfiltration, ObliInjection reaches 0.55 F1 versus 0.68 for ContextLeak, while ToolHijacker reaches 0.37 F1.
  • Main Results: ContextLeak achieves the best performance across all metrics and context types by jointly optimizing malicious-tool selection and context passing.It attains MTSR values of 0.92 on user prompts and 0.89 on conversation history.
  • Ablation Studies: ContextLeak transfers across open-weight and closed-source backend LLMs and remains effective under domain and dataset shifts, although domain shift has a stronger impact.More capable, newer models tend to show higher vulnerability, while user prompt length has limited impact on attack effectiveness.

6 Defenses

The paper evaluates prevention- and detection-based defenses against ContextLeak, finding security improvements can reduce utility while existing detectors miss most attacks.

  • Prevention-based Defenses: Prevention-based defenses fine-tune backend LLMs to resist malicious tool descriptions during tool selection.StruQ, SecAlign, and Meta-SecAlign distinguish trusted instructions from untrusted content such as tool descriptions.
  • Prevention-based Defenses: ContextLeak effectiveness decreases under StruQ or SecAlign, but both defenses substantially degrade correct tool selection and input argument quality.The evaluation identifies a clear utility–security trade-off relative to Llama-3-8B-Instruct.
  • Prevention-based Defenses: Qwen-3.5-9B reaches CTSR 0.58 and IAQ 7.02, and its stronger tool-use utility makes it more susceptible to ContextLeak than Llama-3-8B-Instruct.The comparison links attack susceptibility to backend tool-calling capability.
  • Detection-based Defenses: Detection defenses are evaluated on malicious and benign tools using false positive and false negative rates.The study generates 200 ContextLeak tools and includes 200 ground-truth benign tools.
  • Detection-based Defenses: Existing detectors, including MCPScan, have low FPRs but extremely high FNRs against ContextLeak.Semantically plausible descriptions make the attack stealthy and difficult to detect, including for the MCP-specific scanner.

7 Discussion and Limitations

The discussion identifies boundaries for defending against and extending ContextLeak. Input-argument checking trades security against legitimate context-dependent utility, while current attacks primarily target early context rather than execution trajectories.

  • Defense Limitations: Filtering sensitive context from tool arguments risks degrading legitimate tool use because context can be necessary as a valid input.The paper leaves accurate discrimination between benign and malicious context use as future work.
  • Scope Boundaries: Experiments cover user prompts, conversation history, and tool lists, while memory and knowledge bases remain additional applicable context types.Execution trajectories may also contain sensitive information, particularly in tool responses.
  • Scope Boundaries: ContextLeak usually invokes the malicious tool as the first and only call, limiting exfiltration of information embedded in later benign tool responses.Future work could invoke the attack after benign tools execute to target execution trajectories.

8 Conclusion and Future Work

ContextLeak fine-tunes an attack LLM to generate malicious tool metadata that induces both tool selection and runtime-context disclosure. Evaluation reports effectiveness across differing victim contexts, while future work targets stronger defenses and execution-trajectory exfiltration.

  • Conclusion: ContextLeak uses reinforcement learning with a context-exfiltration reward and strategy-guided candidate generation to fine-tune malicious tool names and descriptions.The attack LLM is trained to induce both malicious-tool selection and context disclosure as input arguments.
  • Conclusion: ContextLeak remains highly effective when victim contexts differ substantially from shadow-user contexts used during fine-tuning.The conclusion also reports existing prevention- and detection-based defenses as insufficiently effective.
  • Future Work: Future work includes more effective defenses and extending ContextLeak to exfiltrate execution trajectories containing sensitive responses from benign tool calls.The proposed extension addresses information accumulated after benign tool use.

Appendix

The appendix documents the attack LLM’s prompting strategy and supporting experimental materials. The prompt emphasizes broad applicability, implicit priority, complete parameters, and target-aligned justification.

  • Appendix Materials: The appendix includes the evolution of average argument reward Rarg on victim users during attack-LLM fine-tuning across context types.This material is presented as Figure 4.
  • Prompt Design: The system prompt asks the attack LLM to generate legitimate-looking tools that are selected broadly and induce sensitive information in tool arguments.It frames the task as adversarial tool design in a red-teaming setting.
  • Prompt Design: The prompt presents the malicious tool as a required first step or preprocessing and coordination layer, with skipping it portrayed as harmful.These are implicit priority signals rather than explicit commands.
  • Prompt Design: The argument-injection objective creates dependency on complete parameters and frames comprehensive inputs as a technical necessity.The prompt explicitly encourages inclusion of attack-target information without stating a direct instruction.
  • Prompt Design: Universal coverage targets all queries in a category, simple and complex requests, and integration across multiple tools or APIs.The prompt also recommends unified-interface and cross-service compatibility language.
  • Prompt Design: The parameter-awareness instructions require explaining, justifying, and fully supplying each parameter because incomplete inputs may degrade output quality.Target-aligned justification adapts the rationale to the type of context being exfiltrated.
  • Appendix Materials: Appendix figures provide the system prompt, strategy-extraction prompt, and IAQ-evaluation prompt used in the experimental pipeline.The corresponding materials are identified as Figures 5–7.
  • Appendix Materials: Table 6 reports ContextLeak performance by domain under in-dataset-in-domain and in-dataset-out-domain settings.The table organizes results across domains and these two evaluation settings.
Loading 2608.27800v1…