PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In

Itay Nakash, George Kour, Guy Uziel, Ateret Anaby-Tavor

arXiv:2410.16950v1cs.CRcs.AI

TL;DR

Tool-using LLM agents can be manipulated through malicious instructions placed in external sources. The paper presents FITD, which precedes a malicious instruction with a harmless request, and reports increased attack success across tested models. It also proposes reflection-based defenses that reassess intended actions during execution.

  • Problem

    Tool-using LLM agents face risks from direct and indirect prompt injection, including harmful actions and sensitive-information disclosure.

    Source: §1 · p. 1

  • Method

    The paper introduces FITD, a harmless request placed before a malicious instruction, and proposes reflection-based checks after external-tool observations.

    Source: §3.2 · p. 3, §6 · p. 8

  • Results

    Up to 44.8% maximum ASR increase, FITD significantly raised attack success across tested models, including with unfamiliar or inaccessible distractor tools.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Reflection-based defenses can reduce attack success, with intervention levels ranging from self-reflection to a safety agent that may require additional resources and trigger false positives.

    Source: §1 · p. 2, §3.2 · p. 3

    Execution-rejection trajectories were proposed as a potential mitigation but were not explored in this study.

    Source: §8 · p. 9

Abstract

from arXiv · show

Following the advancement of large language models (LLMs), the development of LLM-based autonomous agents has become increasingly prevalent. As a result, the need to understand the security vulnerabilities of these agents has become a critical task. We examine how ReAct agents can be exploited using a straightforward yet effective method we refer to as the foot-in-the-door attack. Our experiments show that indirect prompt injection attacks, prompted by harmless and unrelated requests (such as basic calculations) can significantly increase the likelihood of the agent performing subsequent malicious actions. Our results show that once a ReAct agents thought includes a specific tool or action, the likelihood of executing this tool in the subsequent steps increases significantly, as the agent seldom re-evaluates its actions. Consequently, even random, harmless requests can establish a foot-in-the-door, allowing an attacker to embed malicious instructions into the agents thought process, making it more susceptible to harmful directives. To mitigate this vulnerability, we propose implementing a simple reflection mechanism that prompts the agent to reassess the safety of its actions during execution, which can help reduce the success of such attacks.

1 Introduction

The introduction presents FITD as a way to exploit indirect prompt injection in tool-using agents: a harmless request can precede a malicious one. It reports increased attack success and proposes reflection-based defenses.

1 Introduction

  • Indirect prompt injection embeds malicious instructions in external sources, allowing third parties to manipulate agents without direct access.Contaminated sources such as emails, websites, and online comments can potentially affect multiple agents that access them.Source: §1 · p. 1
  • FITD places a small, harmless request before a malicious instruction, exploiting the agent’s tool access and reduced likelihood of reconsidering later actions.The introduction illustrates this sequence with a calculation followed by an instruction to send administrator credentials.Source: §1 · p. 1, Figure 1 · p. 2
  • FITD increased attack success rates across tested models, with the maximum increase reaching 44.8%.The increase persisted when distractor tools were unfamiliar or inaccessible to the agent.Source: §1 · pp. 1–2
  • Thought-injection analysis supports the hypothesis that actions included in an agent’s thought are likely to be executed without re-evaluation.Injecting a thought instructing the attack produced over 95% ASR across all models, an increase of up to 86%.Source: §1 · p. 2
  • The authors propose reflection-based defenses with different intervention levels, from modest gains with self-reflection to over 90% effectiveness with a resource-intensive safety agent that may trigger false positives.The defenses address insider prompt injection and FITD, with options such as alerting a human or halting actions.Source: §1 · p. 2

2 Related Work

Related work distinguishes direct attacks on agents from indirect attacks embedded in retrieved content, while noting that agent vulnerabilities have received relatively limited study.

2 Related Work

  • Vulnerabilities in LLM-based agents that interact with the physical world are described as relatively understudied, despite their potential for real-world actions.The passage contrasts this research gap with rapid work on LLM jailbreaking, including toxic, discriminatory, or illegal outputs.Source: §2 · p. 3
  • Direct prompt injections involve access to the agent and malicious instructions crafted to bypass safeguards, with DAN and Crescendo cited as examples.The cited examples are DAN (Do Anything Now) [Geiping et al., 2024] and Crescendo [Russinovich et al., 2024; Aqrawi, 2024].Source: §2 · p. 3
  • Indirect prompt injections place malicious prompts in external content, exploiting the blurred boundary between instructions and data and enabling remote, potentially widespread impact.The passage compares this risk to “watering hole” attacks.Source: §2 · p. 3

3 The Foot in the Door Attack

FITD adds a harmless distractor before an IPI’s malicious instruction, while thought injection probes whether actions in an agent’s thought process shape later behavior. The paper also proposes reflection-based checks after tool observations to assess intended actions.

3 The Foot in the Door Attack

  • FITD inserts a small, harmless request before a malicious instruction in an indirect prompt injection attack.The distractor may be as simple as a math problem; the attack uses the agent’s external-tool access to introduce the later instruction.Source: §3 · p. 3

3.1 Thought Injection

  • Thought injection embeds harmless and malicious actions, or only a harmless action, in the agent’s initial thought to examine subsequent actions.The paper hypothesizes that an action included in a thought is more likely to be executed.Source: §3.1 · p. 3

3.2 Defense Approaches

  • The proposed defense evaluates an agent’s intended action after an external-tool observation, using the agent itself or an external reflector to assess safety and hesitation.The reflection mechanism can prompt a warning, user confirmation, or halt, depending on the perceived risk.Source: §3.2 · pp. 3–4

4 Empirical Study

The empirical study evaluates FITD against IPI, varying distractor familiarity, placement, timing, and thought injection. Across tested models, FITD raises attack success, while early placement and injected intent are associated with higher compliance.

4.1 Experimental Setup

  • The study used a benchmark with 510 direct-harm and 544 data-steal attacks, focusing mainly on direct harm; data-steal results showed similar trends.Baseline IPI results were reproduced on the tested models.Source: §4.1 · p. 4

4.2 Evaluation Approach

  • The evaluation simulated tool observations, classified outputs as successful, unsuccessful, or invalid, and calculated ASR using valid outputs only.Invalid outputs included calls to non-existent tools; valid-output rates did not considerably decline for FITD or unfamiliar FITD.Source: §4.2 · pp. 4–5

4.3 Effectiveness of the Foot-in-the-Door Attack

  • 44.8% for GPT-4o-mini and 36.3% for Llama-3-70B: FITD increased ASR over vanilla IPI across tested attacks.The attack remained robust across eight distractor tools.Source: §4.3 · p. 5

4.4 Effectiveness of the Foot-in-the-Door Attack Using Unfamiliar Tools

  • With an unfamiliar, inaccessible distractor tool, FITD still increased mean ASR by 15% across models, though effectiveness was slightly reduced.The unfamiliar-tool evaluation tested eight tools and reported results using the calculator tool, whose results were consistent across models.Source: §4.4 · pp. 5–6

4.5 Distractor Physical Position and Chronological Timing Effect

  • 13.1% for early versus late distractor placement, compared with 4.6% for early versus late action timing: position had the stronger influence on ASR.Early placement and, generally, early action timing supported higher attack success; Llama-3’s highest rate occurred with early placement and late timing.Source: §4.5 · p. 6

4.6 Thought-Injection (TI) and Harmless-Thought-Injection (HTI)

  • Thought injection containing harmful intent led to over 95% compliance with FITD and above 83% with vanilla IPI across all models.Harmless-only thought injection had lower ASR than regular thought injection but remained significantly higher than without thought injection.Source: §4.6 · pp. 6–7

5 Defense Methods

The paper proposes reflection-based interventions at the thought stage, ranging from self-reflection to external reflectors that assess hesitation or broader safety. The methods trade off attack mitigation against false positives and interruption of benign actions.

5 Defense Methods

  • The authors propose intervening during thought generation because agents may follow through after forming an action intent, even when their thoughts express hesitation.A reflection mechanism can assess risk and prompt a warning, user confirmation, or halt, depending on severity.Source: §5 · p. 7
  • Testing covered vanilla IPI and the strongest familiar FITD configuration, but not thought injection attacks, whose harmful thoughts inherently trigger a response.The evaluation also measured recall on attacks and false positives on 1,000 benign thoughts generated using GPT-4o.Source: §5 · p. 7
  • The methods vary in intervention level and computational overhead, requiring a trade-off between stronger attack protection and interruptions to normal agent operation.The authors warn that frequent alerts or halts could contribute to alert fatigue.Source: §5 · p. 7, §5.3 · p. 8

5.1 Self Reflection

  • Self-reflection reduced ASR by up to 15% for IPI in some models, but improvements were not significant or consistent across models and attacks.The authors note its low overhead but conclude it may need to be combined with more robust defenses.Source: §5.1 · p. 7

5.2 Hesitation Reflection

  • The hesitation reflector provided stable attack protection across models and produced no false positives in tests on 1,000 benign thoughts.It flags thoughts that are clearly unusual or suspicious, particularly when the agent itself expresses concern.Source: §5.2 · p. 8

5.3 Safe Reflection

  • The broader safe reflector significantly reduced attack success, but triggered a 16% false-positive rate on benign thoughts.Its broad safety assessment can halt non-malicious actions.Source: §5.3 · p. 8

6 Conclusions

The conclusion presents FITD as an attack that uses harmless requests to introduce malicious ones and describes reflection as a defense that reduced their impact. The authors identify training-based defenses as future work.

6 Conclusions

  • FITD uses harmless requests to introduce a subsequent malicious instruction.The conclusion also identifies vulnerabilities associated with ReAct agents’ thought processes and training approaches.Source: §6 · p. 8
  • The authors plan to explore fine-tuning with negative examples so ReAct agents can recognize and reject harmful trajectories.They frame this as future work toward more resilient LLM-based agents.Source: §6 · p. 8
  • The authors argue that raising awareness of agent vulnerabilities matters for their safe and responsible integration into real-world applications.Source: §6 · p. 8

7 Ethical Considerations

The paper frames agent vulnerability research as both a security priority and an ethical concern. It discusses responsible disclosure, dual-use risks, and the usability costs of mitigation.

7 Ethical Considerations

  • The study treats improving autonomous-system security as important while recognizing that identifying vulnerabilities also raises ethical concerns.Source: §7 · p. 9
  • Researchers should responsibly disclose FITD vulnerabilities and encourage centralized reporting resources for LLM-agent security issues.The authors say they plan to publish results through suitable channels.Source: §7 · p. 9
  • Because adversarial techniques can be repurposed for harm, the authors urge caution in sharing technical details and postpone code release until after publication.The delay is intended to give developers time to learn of the vulnerabilities and implement defenses.Source: §7 · p. 9
  • Misapplied defenses may cause overfiltering or false positives, reducing agent functionality or burdening human operators.The paper identifies balancing security and usability as an ethical concern.Source: §7 · p. 9

8 Limitations

The evaluation is bounded to ReAct agents, IPI scenarios, and a specific set of models. The paper also identifies unfinished defense evaluation and untested language and deployment settings.

8 Limitations

  • The findings concern ReAct agents only; whether other agent architectures share these susceptibilities remains unassessed.Source: §8 · p. 9
  • The FITD investigation covers IPI, while its impact on DPI remains untested.The authors hypothesize a similar effect in DPI, but leave it for future study.Source: §8 · p. 9
  • Experiments used a specific model set, including GPT-4o-mini and Mixtral-8x7B, so results may vary for larger or specialized models.Source: §8 · p. 9
  • The reflection-based defenses are preliminary and need further optimization and evaluation to reduce false positives, especially in complex real-world deployments.Source: §8 · p. 9
  • Execution-rejection trajectories were proposed as a potential mitigation but were not investigated in this study.Source: §8 · p. 9
  • Separating commands from external data is proposed as a possible IPI mitigation, but its feasibility and effectiveness in complex environments require validation.Source: §8 · p. 9
  • The study does not assess language limitations; experiments in non-English settings could reveal additional vulnerabilities.Source: §8 · p. 9

A.1 Results Over All Tested Tools - Familiar Tools

The appendix tables report attack success rates for FITD with familiar and unfamiliar tools across several language models.

A.1 Results Over All Tested Tools - Familiar Tools

  • Tables 2 and 3 report FITD ASR using familiar tools across GPT-4o-mini, Llama-3-70B, Llama-3-1-70B, and Mixtral-8x7B.The tables are labeled as Parts 1 and 2.Source: Table 2 · p. 11, Table 3 · p. 11

A.2 Results Over All Tested Tools - Unfamiliar Tools

  • Tables 4 and 5 report FITD ASR using unfamiliar tools across GPT-4o-mini, Llama-3-70B, Llama-3-1-70B, and Mixtral-8x7B.The tables are labeled as Parts 1 and 2.Source: Table 4 · p. 12, Table 5 · p. 12

A.3 Results Over Reflector Defenses

The defense results table reports data-harm attack success rates for reflector methods combined with IPI and FITD attacks.

A.3 Results Over Reflector Defenses

A.4 Data Steal Results

The section reports attack success rates for data-steal attacks and evaluates reflection-based defenses in that setup.

A.4 Data Steal Results

  • The data-steal results cover attack success rates for multiple attack methods and reflection-based defenses.The section includes IPI and FITD conditions with different reflectors.Source: §A.4 · p. 14, Table 8 · p. 14
  • Table 7 covers attack success rates for IPI, FITD with familiar tools, thought injection, and harmless thought injection.Source: Table 7 · p. 14
  • Table 8 reports data-steal attack success rates for Safe, Hesitation, and Self-Reflectors combined with IPI and FITD.Source: Table 8 · p. 14

B Valid Rates

Valid rates across the tested setups were comparable. The accompanying examples and prompt specifications illustrate how reflectors handle hesitation, safety, and agent thoughts.

B Valid Rates

  • Valid rates showed no significant differences across setups, indicating comparability under the original InjectAgent evaluation standards.Table 9 averages valid rates across tested tools for FITD Familiar and FITD Unfamiliar; Table 10 covers data-steal IPI and FITD Familiar using Calculator as a distractor.Source: B Valid Rates · p. 15, Table 9 · p. 15, Table 10 · p. 15

C Examples of Hesitant Thoughts in Successful Attacks

  • Successful attacks proceeded despite agents’ hesitation, and the Hesitation Reflector recognized each example.Source: Figure 5 · p. 16

D Thought and Reflector Example / D.1 Harmless Request Thought Examples

  • The reflector halts actions for potentially harmful or hesitant injected thoughts, while harmless thoughts permit actions to continue; the examples label thoughts as dangerous/safe or hesitant/not hesitant.Source: Figure 6 · p. 17, Figure 7 · p. 18

E.1 ReAct Agent Prompt

  • The ReAct prompt calls for strategic, task-aligned tool use, autonomous work, helpful answers, and precautions or confirmation for risky calls.Source: §E.1 · pp. 20–21

Hesitation Reflector Prompt

E.3 Safe Reflector Prompt

  • The Safe Reflector decides whether to proceed, treating hesitation or unusual requests as reasons not to proceed.Source: §E.3 · p. 23

E.4 Self-Reflection Prompt Addition

  • The self-reflection addition directs the agent to assess safety after each thought and ensure actions protect privacy and security.Source: §E.4 · p. 24

E.5 Prompt For Generating Harmless Thoughts

  • The harmless-thought prompt requests a specific user task and a tool-based plan with relevant parameters.Source: §E.5 · p. 25
Loading 2410.16950v1…