Source-linked AI summary
NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique
TL;DR
LLM performance on autonomous CTF solving has lacked a broad, standardized evaluation. This paper builds a validated open benchmark and automated tool-using framework, finding that GPT-4 performed best overall but with limited success, while model failures varied by capability and context length.
Problem
LLMs’ ability to solve cybersecurity CTF challenges has not been thoroughly evaluated, despite CTFs providing automatically checkable tests of multi-step cyber reasoning.
Source: §1.1 · p. 1Method
The paper constructs a 200-challenge benchmark and an automated framework that uses LLM function calling and external cybersecurity tools.
Source: §1.2 · p. 2, §3 · p. 7, §5 · p. 8Results
GPT-4 performed best overall among five evaluated LLMs, but its success was limited; failure patterns also differed across models and context lengths.
Source: §4.1 · pp. 8–9Takeaways & Limitations
The benchmark and framework provide a public platform for evaluating and improving LLM approaches to interactive cybersecurity tasks.
Source: §1.2 · p. 2, §5 · p. 10The current benchmark contains only validated challenges, excludes categories such as Incident Response, and draws from a single CTF series with imbalance and diversity limitations.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Large Language Models (LLMs) are being deployed across various domains today. However, their capacity to solve Capture the Flag (CTF) challenges in cybersecurity has not been thoroughly evaluated. To address this, we develop a novel method to assess LLMs in solving CTF challenges by creating a scalable, open-source benchmark database specifically designed for these applications. This database includes metadata for LLM testing and adaptive learning, compiling a diverse range of CTF challenges from popular competitions. Utilizing the advanced function calling capabilities of LLMs, we build a fully automated system with an enhanced workflow and support for external tool calls. Our benchmark dataset and automated framework allow us to evaluate the performance of five LLMs, encompassing both black-box and open-source models. This work lays the foundation for future research into improving the efficiency of LLMs in interactive cybersecurity tasks and automated task planning. By providing a specialized benchmark, our project offers an ideal platform for developing, testing, and refining LLM-based approaches to vulnerability detection and resolution. Evaluating LLMs on these challenges and comparing with human performance yields insights into their potential for AI-driven cybersecurity solutions to perform real-world threat management. We make our benchmark dataset open source to public https://github.com/NYU-LLM-CTF/NYU_CTF_Bench along with our playground automated framework https://github.com/NYU-LLM-CTF/llm_ctf_automation.
1 Introduction
The paper addresses limited evaluation of LLMs on autonomous cybersecurity tasks by introducing an open CTF benchmark and automated solving framework. It situates this contribution within prior CTF platforms and early, narrower LLM-based studies.
1.1 Motivation
- CTF challenges test multi-step reasoning and digital action while enabling automatic evaluation through recovered flags.The paper contrasts this with multiple-choice benchmarks such as MMLU[22,49] and GPQA[39].Source: §1.1 · p. 2
- The paper targets autonomous LLM-based CTF solving with access to cybersecurity tools, extending earlier limited studies that often involved human assistance.The motivation connects this goal to prior automated cybersecurity systems and LLM applications in software engineering.Source: §1.1 · p. 2, §1.3 · p. 3
1.2 Contribution
- The proposed benchmark contains 200 public CTF challenges and evaluates LLMs through an automated framework integrated with eight critical cybersecurity tools.The framework analyzes executables, source code, and challenge descriptions.Source: §1.2 · p. 2
- The contributions include an open dataset, automated evaluation with open-source and black-box models, and a toolkit combining cybersecurity tools with function calling.The database and framework are released publicly to support further development of LLM-based cybersecurity methods.Source: §1.2 · p. 2
1.3 Related Work
- Existing CTF platforms primarily support human competitions, whereas this work provides an environment for LLM agents to deploy and solve challenges.Prior studies assessed platform functionality, educational utility, and related CTF uses, but did not provide this LLM-agent environment.Source: §1.3 · pp. 2–3
- Prior LLM-CTF evaluations examined small challenge sets, including seven challenges, 100 PicoCTF problems, or 10 penetration-testing challenges, often without fully automated evaluation.The paper positions its 200-challenge database and five-model testing as broader than these studies.Source: §1.3 · p. 3
2 NYU CTF Bench
NYU CTF Bench is a validated, open benchmark drawn from CSAW competitions, standardized for reproducible deployment and organized across six cybersecurity categories. Its challenges combine metadata, source files, and containerized environments while preserving realistic visibility constraints.
2 NYU CTF Bench
- The benchmark, based on annual NYU CSAW CTF competitions, validates 200 challenges across six distinct cybersecurity categories from an initial database of 568 challenges.Source: §2 · p. 3
- Each challenge was manually checked for setup, software-version robustness, connectivity, configuration completeness, and required files.Server-side challenges were tested across external and internal devices within the same Docker network.Source: §2 · p. 3
- The benchmark uses challenge points from 1 to 500, with qualifying rounds generally easier than finals and difficulty scores subjectively assigned by experienced creators.Figure 1 presents the qualifying-versus-final difficulty distribution.Source: §2 · pp. 3–4, Figure 1 · p. 4
2.1 Benchmark Structure
- The benchmark standardizes challenges from older and newer competitions by validating Docker images, enabling Docker Compose deployment, and integrating environments with source code and metadata.Its directory structure organizes challenges by year, competition, event, category, and challenge name.Source: §2.1 · p. 4, Figure 2 · p. 4
- Each challenge includes visible JSON metadata and, when needed, a docker-compose.yml file, while the ground-truth flag remains hidden from the model.The metadata includes descriptions, files, host information, and ports.Source: §2.1 · p. 4
- Only files listed in challenge.json are exposed to models, while source code, configuration, Dockerfiles, and other files remain available as benchmark references.This visibility rule is intended to mimic real CSAW competition conditions.Source: §2.1 · p. 4
2.2 Benchmark Categories
- The six categories cover cryptography, forensics, binary analysis, reverse engineering, web security, and miscellaneous security tasks with distinct tools and challenge environments.Examples include SageMath for cryptography, forensic analysis tools, debuggers and symbolic execution, web clients, and Android tooling.Source: §2.2 · pp. 5–6
3 Automatic CTF Evaluation Framework with LLMs
The automated framework defines the logic, prompt structures, input specifications, output validation, and error handling needed to reproduce LLM-based CTF solving. Figure 3 summarizes this architecture.
3 Automatic CTF Evaluation Framework with LLMs
- The framework organizes automated CTF solving through underlying logic, procedural steps, prompt structures, validation methods, and error handling.These components are presented as the basis for replication and extension.Source: §3 · p. 6
- The architecture is presented as a five-module framework for coordinating the automated CTF-solving process.The supplied passage introduces the five modules but does not enumerate them.Source: §3 · p. 6
- Figure 3 depicts the architecture of the automated CTF solution framework.The figure is identified as the framework architecture visual.Source: Figure 3 · p. 6
1. Backend Module
The backend module connects the local framework to remote LLM services through supported commercial APIs or deployable open-source model servers. It supports multiple providers and backend configurations.
1. Backend Module
- The backend module supports OpenAI and Anthropic services through API keys and open-source models through TGI or VLLMs endpoints.API keys authorize commercial requests, while open-source deployments expose a URL for receiving model responses.Source: §1 · pp. 6–7
- As of the release date, supported backend configurations included OpenAI models gpt-4-1106-preview, gpt-4-0125-preview, and gpt-3.5-turbo-1106, plus Anthropic models claude-3haiku-20240307, claude-3-sonnet-20240229, and claude-3-opus-20240229.Source: §1 · p. 6
- The open-source backend supports models including Mixtral, DeepSeek Coder, Llama, WizardLM, and Meta-Llama variants deployed locally or through accessible URLs.Users can connect by obtaining a backend URL or deploying models on local servers.Source: §1 · p. 7
2. Data Loader
The data loader supports both containerized challenge servers and local source files, connecting models to challenge resources while cleaning up temporary runtime artifacts.
2. Data Loader
- Docker-based challenges are loaded through Docker Compose, with images pulled from Docker Hub and containers started from configuration files.The framework identifies whether a Docker container exists, then uses challenge metadata to connect through the designated host and port.Source: §2 · p. 7
- Reverse-engineering challenges can load source files into temporary folders mounted inside the player container for analysis.The mounted files provide clues or binaries that the player container can inspect.Source: §2 · p. 7
- A garbage collector stops containers, removes images, and cleans temporary source-file mounts after challenge execution.Source: §2 · p. 7
3. External Tools
The framework augments LLMs with challenge-specific cybersecurity tools and adapts function-calling formats so models can invoke those tools during CTF solving.
3. External Tools
- External tools let LLMs execute commands, create files, disassemble or decompile binaries, verify flags, and stop unproductive attempts.The tools run inside an Ubuntu 22.04 Docker container and use Ghidra for disassembly and decompilation.Source: §3 · p. 7
- Tool availability is tailored to challenge categories, with disassembly and decompilation excluded from categories where they could distract the model.Source: §3 · p. 7
- The formatting module converts prompts into XML or YAML formats, enabling models without native function calling to invoke external tools.OpenAI models natively support function calling, Anthropic models offer partial support, and LLaMA 3 and Mixtral lack the feature.Source: §3 · p. 8
4. Logging System
The logging system records prompts, model outputs, debugging information, tool activity, and solution metadata throughout and after each challenge-solving process.
4. Logging System
- Logs are organized into four Markdown-based categories: system prompts, user prompts, model outputs, and debugging information.Source: §4 · p. 8
- During solving, the system records prompts, responses, and debugging messages in real time for later analysis of model performance and decisions.Source: §4 · p. 8
- After solving, logs are saved as JSON files that can be converted to HTML and include network, challenge, model, and result metadata.Source: §4 · p. 8
5. Prompt Module
The prompt module transforms challenge metadata into structured system, user, and model prompts that guide the LLM from challenge initialization through a solved result.
5. Prompt Module
- The challenge prompt extracts the name, category, host, port, description, and files from challenge.json.Source: §5 · p. 8
- A system prompt states the CTF objective and flag format, while the initial user prompt presents the challenge name, category, description, and files.Source: §5 · p. 8
- The model prompt helps the LLM interpret challenge content and results returned by executed commands until the challenge is marked solved.Source: §5 · p. 8
4 Initial Experiments in Solving CTFs with LLMs
The framework evaluated five LLMs on validated CTF challenges under repeated attempts and 48-hour limits, finding substantial differences across models and categories. Comparisons with human scores and failure types exposed both capability gaps and ethical considerations.
4 Initial Experiments in Solving CTFs with LLMs
- The framework used hosted GPT and Claude APIs alongside open-source models deployed with VLLM and TGI on Nvidia A100 GPUs.The local setup stored model keys and URLs and provided inference endpoints for experiments.Source: §4 · p. 8
- Experiments repeated each challenge five times and counted success when a model solved it at least once within a 48-hour competition-like limit.Incorrect commands, incorrect code, failed flag checks, and exhausted attempts counted as unsuccessful.Source: §4 · p. 8
4.1 Baseline Performance and Comparison with Human CTF Players
- GPT-4 performed best overall, while Claude and GPT-3.5 succeeded in selected categories; Mixtral and LLaMA solved no challenges.The evaluation covered five LLMs across six CTF categories.Source: §4.1 · pp. 8–9, Table 4 · p. 9
- LLM failures comprised connection problems, abandonment, round exhaustion, token limits, and incorrect answers.GPT-3.5 and Claude 3 had high give-up rates, while Mixtral and LLaMA recorded 100% wrong-answer failures; larger-context models had fewer token-limit failures.Source: §4.1 · p. 9
- Claude 3 exceeded the median human score in the 2022 finals with 1500 points, whereas GPT-4’s strongest result was 300 in the 2023 qualifiers.GPT-3.5 scored 500 and 1000 in the 2022 qualifiers and finals but did not score in the 2023 events.Source: §4.1 · p. 9, Table 5 · p. 9
4.2 Ethics Concerning LLMs in Offensive Security
- CTF-based LLM evaluation raises misuse, privacy, security, and legal concerns, yet no experiment involved an LLM refusing a challenge because of ethical conflicts.The authors frame CTFs as educational while noting that misuse risks remain.Source: §4.2 · pp. 9–10
5 Conclusion and Future Work
The paper concludes with a 200-challenge, six-category benchmark and an automated LLM-solving framework, while identifying incomplete validation, category imbalance, and future support needs. Appendix examples and starter software illustrate the surrounding benchmark resources.
5 Conclusion and Future Work
- The benchmark contains 200 validated challenges from seven years of NYU CTF competitions across six categories, but excludes some unvalidated challenges and underrepresents forensics and web.The initial database contained 567 challenges, and Incident Response was omitted because it was difficult to validate.Source: §5 · p. 10
- Future work targets balanced and more diverse data, broader tool and platform support, and model updates aligned with community roadmaps.The current database comes from a single CTF series, and models sometimes select inappropriate tools for the code they analyze.Source: §5 · p. 10
A Some LLM Solutions on CTF Examples
- Figures provide solver excerpts for Puffin, AndroidDropper, a reverse-engineering challenge, and Smug Dino.The reverse-engineering example specifically shows analysis of disassembly and decompilation outputs.Source: Figure 5 · p. 14, Figure 6 · p. 14, Figure 7 · p. 15
B Software Included in our Starter Framework
- The starter framework supplies an Ubuntu 22.04 container with build, networking, reverse-engineering, mathematical, and Python security tools.Included tools include radare2, QEMU, SageMath, pwntools, and gmpy2.Source: B Software Included in our Starter… · p. 15
C Case study for web challenge
The Smug Dino case study examined 31 GPT-3.5 or GPT-4 agent conversations and found that agents often recognized but could not retrieve the hint page. Their fallback behavior ranged from common web scans to more targeted header, redirect, and smuggling analysis.
C Case study for web challenge
- Four of five public solutions documented the hint page, three identified the CVE, and no agent transcript referred to those public solutions.The challenge centered on CVE-2019-20372 and an exploitable NGINX server.Source: C Case study for web challenge · pp. 15–16
- Agents commonly pivoted from the hint page to nikto, sqlmap, XSS, dirb, and gobuster web-exploitation strategies.These tactics included scanning, endpoint enumeration, brute force, and attacks against the recon entry form.Source: C Case study for web challenge · p. 16, Table 7 · p. 16
- Some successful conversations examined headers and link redirects to infer a smuggling attack, while others crafted CURL payloads using challenge-name clues.The transcripts ranged from common exploit techniques to more sophisticated inference-driven tactics.Source: C Case study for web challenge · p. 16
D Whole Challenge List
- The appendix lists 200 challenge instances for each of the crypto, forensics, pwn, reverse-engineering, and web categories, with descriptions combining original metadata and manually interpreted information about each challenge’s intention.It also provides suggested tools for solving the challenges, with Tables 8–12 covering crypto, forensics, pwn, reverse engineering, and web, respectively.Source: D Whole Challenge List · p. 17, Table 8 · p. 17, Table 9 · p. 18