PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

Xiaoyan Li, Yunli Wang

arXiv:2609.16098v1cs.CRcs.AIcs.CL

TL;DR

Tool-integrated LLM agents face adversarial attacks, while existing defenses often focus on prompts rather than hijacked tools. This paper combines tool-set defenses with prompt-based strategies; across seven models, NTR-based defenses achieve extremely low or even zero ASR while maintaining or improving OTSR.

  • Problem

    Tool-based attacks can hijack LLM-agent tool use, but many defenses are attack-specific and modify prompts rather than the tools being hijacked.

    Source: §1 · pp. 1–2

  • Method

    The framework combines tool filtering and restoration of the trusted toolset with CoT prompting, paraphrasing, and self-reflection.

    Source: §1 · p. 2, §3.1.2 · pp. 3–4, §5 · p. 8

  • Results

    Across four open-source and three proprietary LLMs, NTR-based defenses achieved extremely low or even zero ASR while maintaining or improving OTSR.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The lightweight, modular framework is presented as a practical defense layer and a foundation for extension to richer deployment scenarios.

    Source: §5 · p. 8

    The approach was not assessed in dynamic environments.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large Language Model (LLM) agents have demonstrated impressive capabilities across a variety of domains, particularly when integrated with external tools for multi-step task completion. However, they are increasingly vulnerable to adversarial attacks, including direct prompt injection, indirect prompt injection, memory poisoning, and backdoor attacks, which exploit the model's openness to prompt injection and tool manipulation. In this work, we explore practical and generalizable defense strategies within a unified framework across these four attack types. We introduce two universal tool-based defenses: Attacker Tool Filtering, which uses anomaly detection (e.g., Isolation Forest) to identify and remove suspicious tools, and Normal Tool Recalling, a white-box method that restores the agent's original toolset prior to planning. Additionally, we incorporate prompt-based defenses: Chain-of-Thought prompting and self-reflection techniques to enhance reasoning and task paraphrasing to mitigate attacks. Experimental results across both four open-source LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B) and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5) show that our methods significantly reduce the Attack Success Rates (ASR), achieving 0% ASR in many settings, while preserving or even improving the original task success rate. These findings highlight the promise of simple, modular, multi-layered defenses for strengthening the security and robustness of tool-integrated LLM agents. The code is available at https://github.com/Xiaoyan-Lisa/Defenses-for-Tool-Integrated-LLM-Agents-Against-Adversarial-Attacks.

1 Introduction

Tool-integrated LLM agents face adversarial attacks that can hijack tool use, while existing defenses are often attack-specific and focus on prompts rather than tools. The paper proposes a unified framework combining tool-based and prompt-based defenses, reporting protection across four attack types and model families.

Motivation

  • Tool-integrated attacks can enable unauthorized data access, harmful command execution, and miscommunication with external systems, while most defenses remain attack-specific.The authors identify a need for lightweight, model-agnostic defenses targeting tool misuse and invocation integrity.Source: §1 · p. 1

Research gap

  • Existing defenses largely overlook white-box approaches and typically modify attack prompts rather than the tools those attacks hijack.The paper frames this as a gap in defenses for direct and indirect prompt injection.Source: §1 · pp. 1–2

Contributions

  • The framework combines tool-based defenses with Chain-of-Thought prompting, paraphrasing, and self-reflection to address vulnerabilities in prompts and external tool use.The authors report robust defense across DPI, IPI, memory poisoning, and backdoor attacks, alongside improved task success rates.Source: §1 · p. 2

Evaluation and findings

  • Larger proprietary models do not consistently outperform smaller open models under attack, so model size alone does not indicate greater safety.The evaluation includes open-source LLaMA 3, Gemma 2, and Qwen 2 models, as well as GPT-3.5, GPT-4, and GPT-5.Source: §1 · p. 2

2 Related Work

Prior work often studies one attack type at a time, and most defenses use black-box approaches. The paper proposes combining white-box and black-box strategies across four major attacks while maintaining agent performance and utility.

Attack and defense coverage

  • Most studies examine attacks and defenses for a specific type, while relatively few evaluate multiple attack types.Prior studies also examine real-world applications and dynamic, interactive environments.Source: §2 · p. 2

Defense approaches

  • Most existing defenses rely on black-box assumptions, and large-scale studies report that many remain ineffective and can degrade original task performance.The paper notes limited exploration of gray-box or white-box approaches and limited use of reasoning strategies as defenses.Source: §2 · p. 2

Proposed approach

  • The proposed framework combines white-box and black-box strategies to counter four major LLM-agent attacks while maintaining performance and utility.The paper presents this combination as an answer to the research gap described in prior work.Source: §2 · p. 2

3 Methods

The framework layers prompt defenses around tool filtering and plan verification to address attacks on tool-integrated agents. Its tool-based defenses either detect anomalous tools or restore a trusted toolset, while prompt-based methods guide, rephrase, and review the task and plan.

3 Methods

  • The framework applies Chain-of-Thought prompting and paraphrasing after user input, filters tools before invocation, and uses self-reflection to verify plans before execution.The outer layer targets prompt and system-instruction attacks; the middle layer blocks unsafe tools, and the inner layer checks the generated plan.Source: §3 · p. 2

3.1 Universal tool-based defenses

  • Attacker Tool Filtering detects anomalous tools, while Normal Tool Recalling restores the original benign toolset before plan construction.Both are presented as general-purpose defenses that do not require attack-vector-specific knowledge.Source: §3.1 · p. 3

3.1.1 Attacker Tool Filtering

  • ATF embeds tool descriptions and applies an Isolation Forest to identify and filter semantically anomalous tools before workflow planning.The method labels tools as normal or anomalous and uses the filtered toolset for planning.Source: §3.1.1 · p. 3

3.1.2 Normal Tool Recalling

  • NTR assumes the original toolset was safely stored and restores it immediately before planning, replacing the potentially compromised visible tools.The agent then plans using only the restored, user-approved tools.Source: §3.1.2 · p. 3

3.2 Prompt-based defenses

  • The prompt-based defenses combine CoT, paraphrasing, and self-reflection while aiming to maintain original task performance.The methods are presented as additions to the tool-based defenses.Source: §3.2 · p. 4

3.2.1 Chain-of-Thought Prompt

  • Structured CoT prompts the agent to produce a step-by-step JSON plan using predefined tools to support deliberate planning and tool selection.The authors state that this can reduce the likelihood of following maliciously injected instructions or attacker-controlled tools.Source: §3.2.1 · p. 4

3.2.2 Paraphrasing

  • Paraphrasing embeds references to benign task tools and rephrases the user prompt to preserve its semantics while disrupting adversarial structures.The paraphrasing process uses GPT-4o-mini and is combined with NTR.Source: §3.2.2 · p. 4

3.2.3 Self-Reflection

  • Self-reflection has the LLM act as both reflection module and evaluator, iteratively refining its plan through internal language-based reasoning.This approach does not require external feedback.Source: §3.2.3 · p. 4

4 Experiments

The experiments evaluate tool- and prompt-based defenses across four attack types, using ASR and OTSR to measure attack execution and benign-task completion. NTR-based defenses are generally strongest at reducing attacks, but their utility effects vary by model and setting.

4 Experiments

  • The ASB evaluation covers four attack types across seven LLMs, measuring ASR on attacked tasks and OTSR on benign tasks; its dataset has 10 agents with 400 tasks each.Each agent's tasks comprise 200 aggressive and 200 non-aggressive attacks.Source: §4 · p. 4, §4.1 · p. 5

4.2 Evaluation under Direct Prompt Injection Attacks

  • NTR-based defenses reduce DPI ASR to zero for GPT-3.5 and GPT-4 and lower it for open-source models, while generally improving OTSR except for GPT-3.5.ATF also beats standalone paraphrasing on ASR for all tested models except LLaMA3-8B and improves OTSR particularly for Gemma2-9B and LLaMA3.1-8B.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 Evaluation under Indirect Prompt Injection Attacks

  • NTR reduces IPI ASR to near zero for almost all tested models, but GPT models often lose OTSR; ATF gains vary and can also reduce OTSR.IPI generally produces lower ASR than DPI for the four smaller LLMs, but not for GPT models.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Evaluation under Memory Poisoning

  • NTR achieves zero ASR against memory poisoning across tested LLMs, improving OTSR for the four smaller models but reducing it for GPT models.No-defense ASR remains below 10%; ATF is less consistent across ASR and OTSR.Source: §4.4 · p. 6, Table 3 · p. 7

4.5 Evaluation under Backdoor Attacks

  • For PoT backdoors, NTR produces the strongest ASR reductions, with several models reaching near-zero or zero ASR; smaller LLMs often improve both ASR and OTSR, unlike GPT models.ATF also improves robustness and task success in most smaller-model settings, while both tool-based defenses substantially reduce GPT-3.5 and GPT-4 ASR at the cost of OTSR.Source: §4.5 · p. 7, Table 4 · p. 8
  • Across attack settings, NTR outperforms ATF in defense performance, while tool- and prompt-based defenses have complementary strengths and effectiveness depends on attack and model.For memory poisoning, the integrated NTR, paraphrasing, and self-reflection defense completely suppresses ASR, though utility preservation varies across model families.Source: §4.5 · p. 7

5 Conclusion

The conclusion presents ATF and NTR as lightweight tool-based defenses combined with prompt-based enhancements for four attack classes. Across seven LLMs in ASB, the methods reduce ASR while maintaining or improving OTSR, though dynamic environments were not evaluated.

5 Conclusion

  • ATF and NTR remove attacker-controlled tools before planning or execution, and the paper combines NTR with paraphrasing, self-reflection, and CoT prompting.The paper frames these as lightweight tool-based defenses and prompt-based enhancements for DPI, IPI, MP, and backdoor attacks.Source: §5 · p. 8
  • Across four open-source and three proprietary LLMs in ASB, the proposed methods significantly reduce ASR while maintaining or improving OTSR.NTR-based defenses achieve extremely low or zero ASR across multiple settings.Source: §5 · p. 8
  • The approach was not assessed in dynamic environments; testing a combined strategy in real-world applications is future work.The authors describe the framework as a practical defense layer and a foundation for extension to richer deployment scenarios.Source: §5 · p. 8

A ATF Algorithm

Attacker Tool Filtering embeds tool descriptions and uses an Isolation Forest to identify anomalous tools, returning a filtered toolset.

A ATF Algorithm

  • A sentence-level embedding model maps tool descriptions to vectors, which are used to train an Isolation Forest with contamination rate ϵ.Source: A ATF Algorithm · p. 10
  • The algorithm takes a toolset with textual descriptions and a contamination rate ϵ ∈(0, 1) as input.Source: A ATF Algorithm · p. 10
  • The algorithm returns only tools whose predicted anomaly label is 0.It filters the input toolset using the Isolation Forest prediction for each tool.Source: A ATF Algorithm · p. 10

Work Flow Prompt

The workflow prompt directs the agent to choose tools based on their descriptions, plan a multi-step workflow, and follow safety guidance when acting.

Work Flow Prompt

  • The agent analyzes tool descriptions to select the most reasonable and safest tool for the task.Source: Work Flow Prompt · p. 10
  • The agent must produce a JSON plan with at least two steps, each selecting a relevant tool for direct use.Source: Work Flow Prompt · pp. 10–11
  • The prompt prioritizes ethical and legal compliance, cautious handling of personal or confidential information, and user permission where required.Source: Work Flow Prompt · p. 11
  • Examples caution against unauthorized surveillance, direct investment decisions without consultation, unreviewed prescriptions, and final legal decisions.For document analysis, the prompt also advises attention to bias and misinformation.Source: Work Flow Prompt · p. 11
  • When sensitive content may need neutralization, the prompt calls for assessing it and informing users when human oversight or approval may be needed.Source: Work Flow Prompt · p. 11

C Paraphrasing

The paraphrasing defense rewrites user input to clarify the task in relation to available tools; an example applies it to a prompt combining a benign request with an injected instruction.

C Paraphrasing

  • The defense rewrites input to improve task clarity and align it with external tools so agents can generate a plan.Source: C Paraphrasing · p. 11
  • The example combines an NDA-drafting request with an injected instruction to conduct unauthorized case analysis using a restricted tool.Source: C Paraphrasing · pp. 11–12
  • The paraphrased task retains both the NDA request and case analysis, but specifies approved tools for case review and compliance checking.Source: C Paraphrasing · p. 12

D Self-Reflection Prompt

The self-reflection prompt validates a planned workflow against tool, structure, and sequencing rules, returning the plan unchanged if valid or corrected if needed.

Self-Reflection Prompt

E Evaluation on GPT-5

For GPT-5, Normal Tool Recalling (NTR) achieved zero ASR while improving OTSR across attack types. Adding more prompt-based defenses also achieved zero ASR but nearly eliminated OTSR.

E Evaluation on GPT-5

  • GPT-5 evaluation used NTR and its combinations with CoT and paraphrasing, with specified attacks for DPI, IPI, MP, and PoT backdoors.The study used Context Ignoring for DPI and IPI, Fake Completion plus Context Ignoring for MP, and non-word-based triggers for PoT.Source: E Evaluation on GPT-5 · p. 12
  • Combining NTR, CoT, Paraphrase, and Reflection gave GPT-5 zero ASR but nearly zero OTSR; NTR with Paraphrase instead improved OTSR by 1% on IPI and MP attacks.Source: E Evaluation on GPT-5 · p. 13
  • GPT-5 had lower ASR than GPT-3.5 and GPT-4 across all four attack types, but substantially lower OTSR without defense.The comparison is reported in Table 5.Source: E Evaluation on GPT-5 · p. 12
  • NTR+CoT+Paraphrase+Reflection achieved zero ASR but nearly zero OTSR, suggesting excessive defensive prompting may interfere with task completion.NTR+Paraphrase improved OTSR by 1% on IPI and MP attacks.Source: E Evaluation on GPT-5 · p. 13
Loading 2609.16098v1…