PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents can Autonomously Hack Websites

Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, Daniel Kang

arXiv:2402.06664v3cs.CRcs.AI

TL;DR

The paper addresses limited evidence about the offensive capabilities of autonomous LLM agents in cybersecurity. It equips agents with tools, documents, planning, and extended context to test autonomous website hacking, finding that GPT-4 can complete complex attacks without prior vulnerability knowledge while open-source models cannot. The results also include autonomous vulnerability discovery in real-world websites and motivate careful consideration of model deployment and release.

  • Problem

    Little is known about the offensive cybersecurity capabilities of autonomous LLM agents, beyond prior demonstrations of prompting models to generate simple malware.

  • Method

    The paper gives LLM agents browser and function-call access, web-hacking documents, planning, and previous-action context, then evaluates them on website vulnerabilities without human feedback.

  • Results

    GPT-4 hacks 73% of constructed websites, compared with 7% for GPT-3.5 and 0% for all tested open-source models, while the most capable agent finds vulnerabilities in real-world websites.

  • Takeaways & Limitations

    The findings raise questions about widespread LLM deployment and support careful release-policy consideration for frontier models.

  • Takeaways & Limitations

    GPT-4 fails on 3 of 5 hard tasks and 1 of 6 medium tasks, indicating limitations on particularly difficult cybersecurity attacks.

Abstract

from arXiv · show

In recent years, large language models (LLMs) have become increasingly capable and can now interact with tools (i.e., call functions), read documents, and recursively call themselves. As a result, these LLMs can now function autonomously as agents. With the rise in capabilities of these agents, recent work has speculated on how LLM agents would affect cybersecurity. However, not much is known about the offensive capabilities of LLM agents. In this work, we show that LLM agents can autonomously hack websites, performing tasks as complex as blind database schema extraction and SQL injections without human feedback. Importantly, the agent does not need to know the vulnerability beforehand. This capability is uniquely enabled by frontier models that are highly capable of tool use and leveraging extended context. Namely, we show that GPT-4 is capable of such hacks, but existing open-source models are not. Finally, we show that GPT-4 is capable of autonomously finding vulnerabilities in websites in the wild. Our findings raise questions about the widespread deployment of LLMs.

1. Introduction

LLM agents combine tool use, document reading, and recursive context to autonomously hack websites, addressing limited evidence about their offensive cybersecurity capabilities. GPT-4 achieves the strongest reported performance, while component removal and less capable models substantially reduce success.

  • 1. Introduction: The work investigates a gap in evidence about LLM agents’ offensive cybersecurity capabilities beyond prompting models to generate simple malware.Prior work had speculated about cybersecurity offense and defense but had not explored autonomous agents.
  • 1. Introduction: 73.3% (11 of 15) of tested vulnerabilities were hacked by the most capable agent, including complex SQL union attacks without prior vulnerability knowledge.The attack can require 38 actions to extract a database schema, retrieve information, and complete the hack.
  • 1. Introduction: GPT-4 autonomously hacks websites without prior knowledge of the specific vulnerability, enabled by tool use, document access, and previous-action context.The agent can read documents, call browser-manipulation functions, retrieve results, and use prior context.
  • 1. Introduction: 13% success after component removal and 6.7% for GPT-3.5 show that performance depends strongly on the agent components and model capability.Every tested open-source model achieved 0% success.
  • 1. Introduction: Autonomous website hacking costs approximately $9.81 per attempted hack after failures, compared with as much as $80 for human effort.The authors describe these estimates as rough approximations intended to provide cost intuition.

2. Overview of LLM Agents and Web Security

LLM agents are systems that reason, plan, and act through tools, while web vulnerabilities arise in front-end or back-end components. The paper focuses on website vulnerabilities, including SQL injection, while excluding phishing against maintainers.

  • 2. Overview of LLM Agents and Web Security: LLM agents use tools and APIs to act autonomously, then plan and react by feeding tool outputs back as context.Document reading is another useful component, while the paper focuses on tool interaction, planning or reaction, and document access.
  • 2. Overview of LLM Agents and Web Security: Websites typically send front-end requests to back-end servers that may contain sensitive information, so improper access is a security concern.Vulnerabilities can occur in the front-end, back-end, or both.
  • 2. Overview of LLM Agents and Web Security: SQL injection exploits server-side logic that lets user actions send commands to a back-end database, potentially exposing stored information.The paper distinguishes the simple always-true example from the more challenging SQL attacks it tests.
  • 2. Overview of LLM Agents and Web Security: The study considers vulnerabilities in websites themselves and excludes attack classes such as phishing against website maintainers.This defines the scope of the paper’s web-security analysis.

3. Leveraging LLM Agents to Hack Websites

The paper builds autonomous hacking agents around browser and terminal access, web-hacking documents, planning, and carefully designed initial prompts. These agents operate without human feedback and use previous interactions to pursue attacks.

  • 3. Leveraging LLM Agents to Hack Websites: The agent combines function calling, document reading, and planning through the OpenAI Assistants API and LangChain.The implementation uses GPT-series models and programmatic function execution.
  • 3. Leveraging LLM Agents to Hack Websites: Agents access a sandboxed headless browser and terminal tools, enabling programmatic interaction with websites and commands such as curl.The setup does not currently use visual website features.
  • 3. Leveraging LLM Agents to Hack Websites: Six public documents cover general web hacking, SQL injection, XSS, and SSRF without containing instructions specific to the constructed websites.The specific documents are not disclosed because of the work’s sensitivity.
  • 3. Leveraging LLM Agents to Hack Websites: Because agents receive no human feedback, the initial prompt encourages creativity, strategy switching, persistence, and trying new strategies after failure.The prompt is intended to mitigate failures such as getting stuck or abandoning promising strategies.

4. LLM Agents can Autonomously Hack Websites

The evaluation tests autonomous agents on sandboxed real websites containing 15 vulnerabilities across techniques and difficulty levels. Success is defined by achieving each vulnerability’s goal within the execution limit, with five trials per vulnerability.

  • 4. LLM Agents can Autonomously Hack Websites: The evaluation covers 15 vulnerabilities ranging from simple SQL injection to complex combined XSS and CSRF attacks across multiple difficulty levels.The complete vulnerability list and difficulty ratings are reported in Table 1.
  • 4. LLM Agents can Autonomously Hack Websites: Agents are evaluated on sandboxed real websites, and an attack succeeds when its predefined goal is achieved within 10 minutes.The sites retain front-end, back-end, and database functionality while being sandboxed to avoid disrupting real systems.
  • 4. LLM Agents can Autonomously Hack Websites: Five trials are run per vulnerability, with overall success requiring at least one successful trial and pass rate recorded for cost analysis.This reflects the paper’s distinction between one successful cybersecurity attack and conventional repeated-trial metrics.

10. OpenChat 3.5 (Wang et al., 2023a)

The experiments show that autonomous website hacking depends strongly on model capability and agent components, with GPT-4 outperforming GPT-3.5 and open-source models.

  • 10. OpenChat 3.5 (Wang et al., 2023a): 73.3% success was achieved by GPT-4 with document reading, function calling, and the assistant API, without specifying a target vulnerability.The strongest agent succeeded on 11 of 15 vulnerabilities.
  • 10. OpenChat 3.5 (Wang et al., 2023a): GPT-3.5 reached 6.7% success, while every tested open-source model reached 0%.GPT-3.5 correctly executed one SQL injection but failed on the other tasks.
  • 10. OpenChat 3.5 (Wang et al., 2023a): The agent completed complex attacks such as blind SQL injection by extracting a database schema and acting on prior website interactions.The hard SQL union attack required multiple rounds with little to no feedback.
  • 10. OpenChat 3.5 (Wang et al., 2023a): GPT-4 still failed on 3 of 5 hard tasks and 1 of 6 medium tasks, indicating limitations on difficult cybersecurity attacks.Failures included authorization bypass, Javascript attacks, hard SQL injection, and XSS + CSRF.
  • 10. OpenChat 3.5 (Wang et al., 2023a): Removing document reading, detailed system instructions, or both substantially reduced performance, with both removals producing results comparable to GPT-3.5.Removing either documents or the detailed prompt eliminated exploitation of hard vulnerabilities and left few medium vulnerabilities exploited.

5. Understanding Agent Capabilities

GPT-4 can perform complex website attacks by using tools, retaining extended context, adapting to feedback, and planning across multiple attempts, whereas tested open-source models generally cannot complete the exploits.

  • GPT-4 Case Studies: GPT-4 autonomously determines vulnerability types, tests attacks, identifies targets, and executes full exploits using context from previous steps and website feedback.The examples include SQL injection and server-side template injection attacks.
  • GPT-4 Case Studies: GPT-4’s complex attacks can require up to 48 function calls, with SQL union attacks averaging 44.3 actions including backtracking and 38 without it.These attacks require extracting columns and database schemas, retrieving sensitive information, and maintaining that information in context.
  • GPT-4 Case Studies: GPT-4 achieves 100% success on the reported SQL injection and CSRF vulnerabilities, while harder vulnerabilities have lower success rates.The authors note that a single successful attack can achieve an attacker’s goal, so even a 20% success rate may matter operationally.
  • Open-source LLMs: The tested open-source models fail to complete the exploits, with failures attributed largely to incorrect tool use and inadequate planning.The result includes large models and models tuned on more than 1,000,000 GPT-4 examples.
  • Open-source LLMs: OpenChat-3.5 detects the correct vulnerability 25.3% of the time but fails to use website feedback to perform the attack, whereas GPT-4 adapts its strategy.OpenChat-3.5 nevertheless is described as the most capable tested open-source model despite having 7 billion parameters.
  • Open-source LLMs: The authors suggest that further tuning could make open-source models capable of hacking websites and motivate discussion about responsible release.This is presented as a forward-looking implication of the observed model differences.

6. Hacking Real Websites

The study deployed its most capable agent on approximately 50 curated real websites and found that GPT-4 autonomously identified an XSS vulnerability on one site.

  • Hacking Real Websites: GPT-4 found an XSS vulnerability on 1 of approximately 50 curated real websites.The tested sites were selected using criteria intended to identify potentially vulnerable, older, and possibly unmaintained websites.
  • Hacking Real Websites: The finding demonstrates that GPT-4 can autonomously identify vulnerabilities in real-world websites, although the identified site did not record personal information and no concrete harm was found.The website identity was withheld while the researchers attempted responsible disclosure.

7. Cost Analysis

The cost analysis estimates that autonomous GPT-4 hacking can be substantially cheaper than human expert effort, while emphasizing that the estimates are rough and intended mainly to provide intuition.

  • Cost Analysis: At a 42.7% overall success rate, GPT-4 hacking costs approximately $9.81 per website after accounting for failed attempts.The estimate comes from five runs with an average token cost of $4.189.
  • Cost Analysis: The authors highlight that autonomous agents need not know the vulnerability beforehand, can test vulnerabilities in parallel, and have declining costs over time.These features are presented as reasons autonomous hacking could become economically feasible.
  • Cost Analysis: The estimated cost of human expert effort is approximately 8× greater than using the LLM agent.The comparison is based on an estimated 20 minutes for an analyst to check a website for a vulnerability and an assumed salary.
  • Cost Analysis: The cost estimates are rough approximations intended to provide intuition rather than exact measures of hacking costs.A full analysis would require understanding the internal operations of black-hat organizations, which is outside the study’s scope.

8. Related Work

Prior work has examined LLMs in cybersecurity, LLM safety, and internet security, but the paper positions its contribution as a systematic study of autonomous LLM-agent website hacking.

  • Related Work: Existing cybersecurity research has considered LLM-assisted offense and defense, including malware creation and scalable spear-phishing, but the paper identifies autonomous website hacking as understudied.The cited work spans speculation about offense and defense and studies of LLM-supported cyberattacks.
  • Related Work: Research on LLM security focuses on bypassing safeguards through jailbreaking and fine-tuning away RLHF protections.The paper describes these studies as complementary because vendor blocking could motivate attempts to bypass protections.
  • Related Work: Internet security research is broad, and the paper emphasizes that website hacking can enable theft of private information, blackmail or ransomware, and deeper system penetration.The paper argues that automating website hacking could reduce attack costs and increase prevalence.

9. Conclusion and Discussion

The paper shows that LLM agents can autonomously hack websites without prior vulnerability knowledge, with capabilities varying sharply across model families. These findings motivate careful consideration of frontier-model deployment and release policies.

  • GPT-4 hacked 73% of constructed websites, compared with 7% for GPT-3.5 and 0% for all tested open-source models.
  • The agents could autonomously find vulnerabilities in real-world websites without knowing the vulnerability ahead of time.
  • The results show a need for LLM providers to carefully consider deploying and releasing frontier models.

Impact Statement and Responsible Disclosure

The authors acknowledge that their results could enable illegal black-hat attacks, while limiting testing to sandboxed websites and withholding detailed reproduction instructions. They also disclosed the findings to OpenAI before publication.

  • The results could potentially be misused for illegal black-hat hacking of real-world websites.
  • The experiments used sandboxed websites to avoid affecting real-world systems or violating laws.
  • The authors withheld detailed attack steps because they judged the potential downsides of public release to outweigh its benefits.
  • The findings were disclosed to OpenAI before publication.
Loading 2402.06664v3…