Teams of LLM Agents can Exploit Zero-Day Vulnerabilities

• HPTSA extends single-agent approaches with a hierarchical planner, team manager, and task-specific agents for exploring and attempting different vulnerability types.The planner explores websites, determines targets, and dispatches specialized agents through the team manager.Source: §1 · p. 1
• The benchmark contains 14 recent, reproducible real-world vulnerabilities spanning different types and severities and beyond GPT-4’s stated knowledge cutoff.Source: §1 · p. 1
• 42% pass at 5 put HPTSA within 1.8× of GPT-4 with vulnerability knowledge on the real-world zero-day benchmark.Source: §1 · p. 1
• On the benchmark, HPTSA achieves a pass at 5 of 42%, remaining within 1.8× of a GPT-4 agent with knowledge of the vulnerability.Source: §1 · p. 1
• Open-source vulnerability scanners achieved 0% on the benchmark, and expert agents were necessary for high performance.Source: §1 · p. 2

2.1 Computer Security

• Zero-day vulnerabilities are unknown to the deployer, unlike disclosed but unpatched one-day vulnerabilities known to the attacker.Source: §2.1 · p. 2
• Zero-day vulnerabilities are particularly harmful because deployers cannot proactively mitigate them, and web vulnerabilities can provide an initial attack surface.Source: §2.1 · p. 2
• A vulnerability class can be known while a specific instance remains novel, making discovery of particular zero-day instances critical.Source: §2.1 · p. 2

2.2 AI Agents and Cybersecurity

• Tool-enabled LLM agents use APIs to act on tasks, and prior ReAct-style agents exploit toy or one-day vulnerabilities when given descriptions.Source: §2.2 · p. 2
• Prior agents fare poorly in zero-day settings because long-term exploration expands context and makes backtracking across vulnerability types difficult.Source: §2.2 · p. 2

3 HPTSA: Hierarchical Planning and Task-Specific Agents

• HPTSA targets rapidly growing cybersecurity context and difficult backtracking by using multiple agents with hierarchical planning and task specialization.Source: §3 · p. 2

3.1 Overall Architecture

• The hierarchical planner explores websites and identifies instructions or targets, while the team manager selects agents and reuses prior traces to refine attacks.Source: §3.1 · p. 3

3.2 Task-Specific Agents

• HPTSA uses six expert agents for XSS, SQLi, CSRF, SSTI, ZAP, and generic web hacking, with tools, documents, and tailored prompts.Source: §3.2 · p. 3
• The study limits its task-specific-agent investigation to web vulnerabilities and leaves applicability to code scenarios for future work.Source: §3.2 · p. 3

3.3 Implementation

• The implementation connects agents through LangGraph and reduces costs by simplifying client-side HTML before passing it to agents.Source: §3.3 · p. 3

• Vulnerabilities were selected beyond GPT-4’s knowledge cutoff to reduce training-data leakage in the zero-day evaluation setting.Source: §4 · p. 3
• The benchmark focuses on web vulnerabilities with specific triggers and clear pass-or-fail measures, making exploitation outcomes easier to evaluate.Source: §4 · pp. 3–4
• Table 1 lists each vulnerability and description, while Table 2 provides CVE numbers, publication dates, and severity ratings.Source: Table 1 · p. 4, Table 2
• All included vulnerabilities were manually exploitable and reproducible, with testing performed in a sandboxed environment to avoid harming real users.Source: §4 · p. 4
• The benchmark contains 14 web vulnerabilities spanning XSS, CSRF, SQLi, arbitrary code execution, and other types, all with medium-or-higher severity.Source: §4 · p. 4

5.1 Experimental Setup

• The evaluation measures manually verified exploitation success using pass at 5 as the primary metric because one successful attempt is sufficient to compromise a system.Source: §5.1 · p. 4
• The setup compares proprietary and open-source models, one-day and no-description agents, open-source scanners, and HPTSA variants while tracking token-based dollar costs.Source: §5.1 · pp. 4–5

5.2 End-to-End results

• HPTSA with GPT-4 achieves 42% pass at 5 and 18% pass at 1, while outperforming GPT-4 without descriptions by 2.0× and 4.3×, respectively.Source: §5.2 · p. 5, Figure 3 · p. 5
• HPTSA performs within 1.8× of the one-day GPT-4 agent with vulnerability descriptions on pass at 5, while ZAP and MetaSploit achieve 0%.Source: §5.2 · p. 5, §5.1 · p. 5

5.3 Ablation studies

• Removing task-specific agents lowers pass at 1 by 2.1× and pass at 5 by 50%, while removing documents lowers them by 2.1× and 20%, respectively.Source: §5.3 · p. 6, Figure 4 · p. 6
• When task-specific agents are used randomly without hierarchical structure, pass at 1 is 13× lower and pass at 5 is 6× lower, demonstrating hierarchical structure’s necessity.Source: §5.3 · p. 6

6.1 Success Case Studies

• HPTSA successfully exploited flusity-CMS CSRF and XSS vulnerabilities by navigating relevant administrative pages and crafting working payloads.The CSRF agent created a menu and reproduced the action in a payload, while an XSS agent exploited the gallery addOn vulnerability.Source: §6.1 · pp. 6–7
• Backtracking and retries help HPTSA reach vulnerability-specific pages, although several runs still fail to navigate to the required location or produce a working payload.This behavior is illustrated for CVE-2024-34061 and the flusity-CMS post-creation attempts.Source: §6.1 · p. 7
• HPTSA synthesizes execution traces to redirect later agents toward promising pages, including /admin.php and specific vulnerability targets.The supervisor uses failed and successful traces to refine task-specific instructions and focus subsequent exploration.Source: §6.1 · p. 7

6.2 Unsuccessful Case Studies

• HPTSA cannot exploit CVE-2024-25635 because its required API endpoint is absent from the website and public documentation unavailable to the agent.The general agent was unable to discover the endpoint despite the presence of a fallback agent for vulnerabilities outside the expert-agent set.Source: §6.2 · p. 7
• HPTSA also fails on CVE-2024-33247 because the required route is difficult to discover and lacks visible input fields for SQL injection.The authors suggest forcing expert agents to explore less accessible endpoints as a possible improvement.Source: §6.2 · p. 7

7 Cost Analysis

• $4.39 was the average GPT-4 cost per run, corresponding to $24.4 per successful exploit at an 18% overall success rate.The per-run cost was comparable to the one-day setting, while the total cost was 2.8× higher.Source: §7 · p. 7
• GPT-4 cost 3.1–15× more per run than the evaluated open-source models, which failed to resolve any tasks.Costs were estimated from input and output token usage using model-specific API prices.Source: §7 · p. 7
• $75 was the estimated cost for a human cybersecurity expert spending 1.5 hours exploring a website at $50 per hour.The authors characterize this estimate as higher, but not dramatically higher, than using an AI agent.Source: §7 · p. 8
• Projected cost reductions could make GPT-4o-level agents 3–6× cheaper than today within 1–2 years if current trends continue.The paper cites GPT-4o price reductions and Claude-3.5-Haiku’s lower input-token cost as examples of this trend.Source: §7 · p. 8

8 Related Work

• Prior ReAct-style agents can hack toy websites and described real-world vulnerabilities but perform poorly when vulnerability descriptions are withheld in the zero-day setting.The paper identifies difficulty backtracking from dead ends as a key challenge.Source: §8 · p. 8
• Human-uplift research studies how LLMs assist penetration testing and malware generation, while related work examines broader societal implications for cybersecurity.These lines of work concern human-supported cybersecurity and possible societal effects rather than autonomous zero-day exploitation.Source: §8 · p. 8
• This work differs by introducing a real-world AI-agent system based on hierarchical planning and task-specific agents.The authors position it as the first such system to their knowledge, building on prior multi-agent research.Source: §8 · p. 8
• Security-of-agent research studies bypasses, malicious attacks, and indirect prompt injection against AI agents, which the paper describes as orthogonal to its focus.The distinction is between securing agents and using agents to exploit computer-system vulnerabilities.Source: §8 · p. 8

9 Conclusions

• Teams of LLM agents can autonomously exploit zero-day vulnerabilities, resolving an open question posed by prior work (Fang et al., 2024a).The authors frame this as evidence that AI-agent capabilities will increase the pace of cybersecurity activity.Source: §9 · p. 8
• AI agents may support both black-hat website attacks and more frequent penetration testing, leaving their relative benefit to offense versus defense unclear.The authors identify this balance as an open direction for future work.Source: §9 · p. 8

• The authors emphasize that substantial work remains to understand the implications of AI agents in cybersecurity.Source: §10 · p. 8
• The focus on web, open-source vulnerabilities may produce a biased sample, limiting how broadly the findings apply.The authors call for broader investigation of AI-agent implications in cybersecurity.Source: §10 · p. 9
• The authors do not publicly release code or prompts to reduce the risk of malicious use.They state that the agents remain confidential at OpenAI’s request and report disclosing findings to OpenAI.Source: §10 · p. 9