Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks

1. Introduction

• Mantis embeds crafted prompt injections in system responses to redirect LLM-agents and disrupt their attack strategies in real time.The framework repurposes LLM susceptibility to adversarial inputs as a defense.Source: §1 · pp. 1–2
• Decoy services attract LLM-agents early in the attack chain, while Mantis dynamically selects countermeasures based on detected interactions.Examples include fake FTP servers and compromised-looking web applications.Source: §1 · p. 2
• Mantis supports both disruption and hack-back, steering attackers toward actions that can compromise their own systems.The injections can be hidden from human operators using ANSI escape sequences and HTML comment tags.Source: §1 · p. 2
• Over 95% efficacy across diverse configurations was reported for Mantis in simulated attack scenarios.The evaluation used ChatGPT-4, ChatGPT-4o, Claude3.5-Sonnet, and Claude3.5-Haiku.Source: §1 · p. 2
• The experiments were restricted to controlled environments, including local sandboxes or machines configured for penetration testing.Attacker systems ran in VMs without internet access except for essential secure channels.Source: §1 · p. 2

2.1. Prompt Injection

• Indirect prompt injection targets external resources accessed by an LLM, and Mantis uses this approach as a context-specific defensive strategy.Direct injection instead supplies manipulated input through interfaces such as chatbots or API endpoints.Source: §2.1 · p. 2

2.2. LLM-agents and Automated Cyberattacks

• An LLM-agent combines an instruction-tuned model with an interaction framework, planning actions, executing them, and refining strategy from feedback.Its available tools form its action space; the task is formalized as (objA, env).Source: §2.2 · pp. 2–3
• Cyberattack LLM-agents operate across stages from reconnaissance to exploitation, and the paper frames defense against agents spanning the cyber kill chain.The preliminaries situate these agents among prior attack frameworks and penetration-testing work.Source: §2.2 · p. 3

3. Threat Model

• The attacker is an LLM-agent seeking to compromise a remote target from knowledge limited to its IP address, while completing the cyber kill chain.Its objective may include opening a shell or exfiltrating sensitive information.Source: §3 · p. 3
• The defender operates on the target without knowing the attacker’s strategy, LLM, objective, or the target’s vulnerabilities in advance.It therefore cannot patch those vulnerabilities before the attack.Source: §3 · p. 3
• The defender’s sabotage objective may compromise the attacker’s machine or stall the agent, and the defender wins only if the attacker fails and sabotage succeeds.The attacker is allowed at most nmax actions.Source: §3 · p. 3

4. Mantis: Overview and Architecture

• Mantis embeds prompt injections in system responses because those responses inform the LLM-agent’s next actions.The system-agent interaction is treated as a communication medium between defender and agent.Source: §4 · p. 3
• Decoys attract agents to intentionally vulnerable services, confirm malicious intent, and deliver injections crafted by the injection manager.The manager coordinates real-time injection deployment and counterstrike operations, including spawning a tarpit or reverse-shell listener.Source: §4 · p. 4

4.1. Sabotage Objectives

• The active agent-counterstrike objective manipulates the agent into opening a reverse shell on its host, compromising the attacker’s machine.The defender may then gather intelligence, disrupt the campaign, or use the machine as an access point to the attacker’s network.Source: §4.1 · p. 4
• The passive agent-tarpit objective traps an agent in inconsequential actions, slowing attacks and consuming attacker resources.Mantis can inflate the data processed per round, wasting GPU time or increasing costs, particularly for API-based proprietary LLMs.Source: §4.1 · p. 4

5. Decoys: Fake Services that Attract Agents

• Deploying injections after decoy exploitation verifies malicious intent and makes it easier to redirect an agent that has already progressed toward its attack objective.The decoy sends the payload in a subsequent response after exploitation.Source: §5 · p. 4

5.1. Decoy Instantiations

• Mantis uses frequently targeted FTP and web-app decoys, including anonymous FTP access and SQL-injection-prone login fields, to engage agents before they reach real systems.The paper also describes the framework as modular and adaptable to other services or existing honeypots.Source: §5.1 · p. 5
• The FTP decoy injects after anonymous login, with get or ls as fallback; the web decoy injects after SQL-injection authentication bypass or database dumping.Source: §5.1 · p. 5
• The selected decoys reflect frequently misconfigured systems represented in CTF resources used by LLMs, aiming to exploit those agents’ learned biases.The paper describes this as creating a “supernormal stimulus” to draw agents toward decoys.Source: §5.1 · p. 5

5.2. Hiding Payloads from Regular Users

• ANSI escape sequences hide terminal payloads from people while remaining in raw output processed by agents; HTML comments combined with ANSI conceal payloads in browsers and curl.The paper says hiding payloads also reduces the chance that hands-on-keyboard attackers identify the decoy during initial reconnaissance.Source: §5.2 · pp. 5–6

5.3. Decoy Banner Manipulation

• Hidden banner prompts can suggest a vulnerability and steer an agent toward a decoy and away from legitimate services, although they are not intended for direct execution.Source: §5.3 · p. 6

6.1. Execution Triggers

• Intent-aligned natural-language triggers are more effective than traditional “ignore previous instructions” triggers because they capitalize on the agent’s attack momentum.Mantis uses brief, randomly selected triggers across sabotage objectives and decoy services.Source: §6.1 · p. 6

6.2. Active Defenses (agent-counterstrike)

• For agent-counterstrike, a decoy login activates an injection that directs the agent to fetch and blindly execute a reverse-shell initiator.Mantis starts an HTTP server and listener on random free ports, then tests whether the resulting shell is functional.Source: §6.2 · p. 7
• Active counterstrike raises ethical and legal concerns, and attacker-side sandboxing can limit its impact.The authors note that a heavily sandboxed hosting machine may constrain arbitrary-code execution.Source: §6.3 · p. 7

6.3. Passive Defenses (agent-tarpit)

• The passive tarpit routes an agent from an entry-point decoy into a fake filesystem, where injections urge it to search for nonexistent target data.FTP is used to restrict the agent’s action space; dynamically generated directories and repeated injections sustain exploration.Source: §6.3 · pp. 7–8
• In experiments, agents inside the tarpit remained in depth-first searches for nonexistent files and did not backtrack or abandon the path.The authors report that tested LLM-agents continued navigating through inconsequential actions.Source: §6.3 · p. 8
• Large human-like files were an unreliable resource-exhaustion tactic because downloading them could require leaving FTP, while thousands of fake directories could fill the model’s context window.The directory-heavy alternative was described as unrealistic and likely to raise human suspicion.Source: §6.3 · p. 9

7.1. Implementing Attacker’s LLM-agents

• The evaluation addresses limited public agent implementations by using available open-source agents and state-of-the-art OpenAI and Anthropic backend models.The paper identifies PentestGPT, AutoPenAgent, and HackingBuddyGPT as the publicly available solutions considered.Source: §7.1 · p. 9

7.1.1. PentestGPTAuto.

• PentestGPTAuto adds a weaponizer that translates PentestGPT’s natural-language plans into executable commands and feeds command outputs back for further planning.The authors leave PentestGPT’s reasoning and planning modules unchanged.Source: §7.1.1 · pp. 9–10

7.1.2. AutoPenAgent.

• AutoPenAgent is used with its original core behavior, with code extended to support Claude models alongside OpenAI models.The agent uses a ReAct framework and scratchpad-like memory.Source: §7.1.2 · p. 10

7.1.3. HackingBuddyGPT.

• HackingBuddyGPT is adapted for end-to-end attacks by adding local code execution and interactive shell sessions, without changing its planning or decision-making.The evaluation could not add support for Anthropic models.Source: §7.1.3 · p. 10

7.1.4. Other Agents.

• AutoAttacker and PenHeal were not available for testing because their authors could not share the implementations at that time.The authors indicated a release might be possible in the future; all tested agents use a Kali Linux machine to execute commands.Source: §7.1.4 · p. 10

7.2. Implementing the Defender’s Machines

• The CTF flag provides a binary success measure, and a forward-proxy-like server runs Mantis while forwarding traffic to HackTheBox-hosted targets.The chosen machines could not be run on-premise.Source: §7.2 · p. 10
• The defender’s targets are three very-easy HackTheBox CTF machines, selected as a challenging case for Mantis because agents can exploit them readily.The paper reports that PentestGPTAuto achieved close to 95% success on these machines without Mantis and says Mantis was more effective on harder-to-exploit systems.Source: §7.2 · p. 10

7.3. The Setup of the Experiments

• Each experiment configures one decoy and either counterstrike or tarpit sabotage; the defender wins only if the attacker fails to capture the flag and the sabotage objective succeeds.Attacks are capped at 30 rounds, and defender victory requires a functional reverse shell for counterstrike or reaching the round limit while in the tarpit.Source: §7.3 · p. 11

8.1. Attacking Without Mantis Protection

• Without defense, agents generally exploited targets reliably; PentestGPTAuto alone consistently solved CTF:Dancing, while all agents solved the two less complex CTFs.Many tested machines had weak authentication, enabling completion within 4 to 6 rounds.Source: §8.1 · pp. 11–12

8.2. Attacking When Mantis Defends

• Counterstrike was most reliable because opening a reverse shell immediately met its objective, while FTP outperformed Web-app and failed Web-app attempts could divert agents to vulnerable SMB on Dancing.The authors identify repeated malformed SQL-injection commands as a primary cause of sabotage failure with the Web-app decoy.Source: §8.2 · pp. 12–13
• Across all possible configurations, Mantis achieves approximately a 95.4% success rate in achieving its sabotage objective objD and reduces the attacker’s success rate to less than 3%, as summarized in Figure 10.Source: §8.2 · p. 12

8.3. Resource draining agent-tarpit

• Increasing tarpit filesystem complexity raises per-attack GPT-4o API cost through larger prompts, while agent memory and prompting designs shape cost differences.Figure 11’s cost experiment uses the FTP decoy on CTF:Dancing and halts after 10 tarpit iterations; the authors note that higher limits could increase cost further.Source: Figure 11 · p. 13, §8.3 · p. 13

• Mantis embeds context-specific prompt injections in interactions between vulnerable systems and LLM agents to disrupt adversarial agents.Source: §9 · p. 13
• Mantis’s success as a prompt-injection defense depends largely on whether attackers can modify their LLMs to avoid prompt injection.The authors describe prompt injection as one of the most difficult challenges in LLM security.Source: §9 · p. 13
• Whether defenses tailored to LLM-agent contexts can counter prompt injection, rather than relying on broader Generative AI defenses, remains an open problem.Source: §9 · pp. 13–14
• Attackers who learn Mantis’s defenses can adjust tactics to bypass decoys or filter execution triggers, potentially requiring human guidance of attacking agents.The authors argue this added unpredictability increases the operational costs of automated cyberattacks.Source: §9 · p. 14

Appendix A. Testing on more complex CTFs

• Beginner-level CTFs were chosen as a best-case setting because agents had a genuine chance of success, making Mantis’s interventions measurable.A successful defense represented an attack the agent would have completed without Mantis.Source: Appendix A · p. 16
• On harder CTFs, current agents lacked the multi-step reasoning and exploit sophistication to reach successful exploitation reliably without human guidance.The authors therefore considered these tasks less meaningful for measuring Mantis, since failure could stem from challenge complexity rather than defense.Source: Appendix A · p. 16
• 0% of five unaided trials on each of Chemistry and Cicada completed the exploit within the 30-round limit.The tested agent was PentestGPTAuto with GPT-4-o.Source: Appendix A · p. 16
• 100% of the tested complex-CTF runs with Mantis active misdirected the agent to an FTP decoy.Tests used the agent-counterstrike objective.Source: Appendix A · p. 16

Appendix B. Additional Results

• Appendix B presents additional results alongside those reported in Section 8.Source: Appendix B · p. 16
• The appendix’s stated role is to complement the results in Section 8.Source: Appendix B · p. 16
• The passage introduces the appendix as supplementary to the paper’s Section 8 results.Source: Appendix B · p. 16

B.1. Evaluation on additional LLMs

• Table A.1 reports PentestGPTAuto results using ChatGPT-4 and Claude3.5-Haiku as base LLMs.The table abbreviates these models as GPT-4 and Haiku3.5, respectively.Source: §B.1 · p. 16
• The appendix focuses on PentestGPTAuto, identified as the most performant among the tested agents.Source: §B.1 · p. 16
• The authors report that results with these additional base LLMs align with those reported for other base LLMs in Section 8.Source: §B.1 · p. 16

B.2. Additional combination of Decoys and Sabotage objective

• The Web-app decoy plus agent-tarpit sabotage combination was omitted from Table 1 and is reported separately in Table A.2.The results focus on PentestGPTAuto.Source: §B.2 · p. 16
• Mantis’s success rate with this combination remains consistent with the Web-app decoy results in Table 1.Source: §B.2 · p. 16
• Reaching the tarpit takes more actions on average because the agent must jump from the Web-app decoy to the FTP-tarpit server.The increased action count is reflected in the reported average number of rounds.Source: §B.2 · p. 16