PaperlayerView exact revision on arXiv

Source-linked AI summary

CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, Dawn Song

arXiv:2506.02548v3cs.CRcs.AIcs.LG

TL;DR

Existing cybersecurity evaluations are limited by small scale and static outcomes, leaving a need for broader assessment of real-world security capabilities. CyberGym addresses this gap with a large-scale benchmark in which agents reproduce vulnerabilities from descriptions and codebases, and its evaluation finds low success rates alongside real-world vulnerability discoveries.

  • Problem

    Existing cybersecurity benchmarks are small-scale and focused on static outcomes, limiting coverage of practical cybersecurity complexity and the evolving security landscape.

  • Method

    CyberGym evaluates agents on generating executable proof-of-concept tests from vulnerability descriptions and corresponding codebases, using 1,507 instances across 188 open-source projects.

  • Results

    CyberGym is challenging for current agents: the top-performing agent-model combination achieves a 22.0% success rate, while the benchmark also uncovers 34 zero-day vulnerabilities.

  • Takeaways & Limitations

    CyberGym provides a benchmark for tracking AI cybersecurity progress and a platform for direct real-world security impact through vulnerability discovery.

  • Takeaways & Limitations

    CyberGym primarily covers memory-safety vulnerabilities in C/C++ projects because it relies on sanitizers for detection.

Abstract

from arXiv · show

AI agents have significant potential to reshape cybersecurity, making a thorough assessment of their capabilities critical. However, existing evaluations fall short, because they are based on small-scale benchmarks and only measure static outcomes, failing to capture the full, dynamic range of real-world security challenges. To address these limitations, we introduce CyberGym, a large-scale benchmark featuring 1,507 real-world vulnerabilities across 188 software projects. Adjustable to different vulnerability analysis settings, CyberGym primarily tasks agents with generating a proof-of-concept test that reproduces a vulnerability, given only its text description and the corresponding codebase. Our extensive evaluation highlights that CyberGym effectively differentiates agents' and models' cybersecurity capabilities. Even the top-performing combinations only achieve a ~20% success rate, demonstrating the overall difficulty of CyberGym. Beyond static benchmarking, we show that CyberGym leads to the discovery of 34 zero-day vulnerabilities and 18 historically incomplete patches. These results underscore that CyberGym is not only a robust benchmark for measuring AI's progress in cybersecurity but also a platform for creating direct, real-world security impact.

1 INTRODUCTION

Existing cybersecurity evaluations are limited by small scale and static outcomes, motivating CyberGym’s large, realistic benchmark and execution-based assessment. The benchmark differentiates current agents while also supporting real-world vulnerability discovery.

  • Motivation: Existing benchmarks can be small-scale and static, limiting coverage of practical cybersecurity complexity and the evolving security landscape.The paper identifies these as two limitations of prior evaluations.
  • CyberGym: A Large-Scale, Realistic Cybersecurity Benchmark: CyberGym contains 1,507 benchmark instances from real-world vulnerabilities across 188 diverse software projects.Instances are quality-controlled through automated filters and manual validation.
  • CyberGym: A Large-Scale, Realistic Cybersecurity Benchmark: Agents receive a vulnerability description and associated codebase, then generate a proof-of-concept test that reproduces the vulnerability.Generated PoCs are executed on pre-patch and post-patch versions to validate reproduction success.
  • CyberGym Challenges Frontier Agents with a Ladder of Difficulty: The evaluation covers four agent frameworks and eleven frontier LLMs, using CyberGym to distinguish cybersecurity capabilities across approaches.CyberGym is designed as a challenging benchmark with tasks spanning different difficulty levels.
  • CyberGym Challenges Frontier Agents with a Ladder of Difficulty: 22.0% is the highest reported success rate for an evaluated agent-model combination, while specialized software engineering models achieve no more than 2.0%.The analysis finds that current approaches mainly solve simpler tasks requiring fewer execution steps and shorter PoCs.
  • CyberGym Extends to Creating Direct, Real-World Security Impact: CyberGym extends beyond benchmarking by uncovering 17 inadequate historical patches and 34 zero-day vulnerabilities.The zero-days include 10 found during vulnerability reproduction and 25 from open-ended discovery across 431 open-source projects.

2 RELATED WORK

Prior cybersecurity benchmarks use either idealized capture-the-flag tasks or smaller collections of historical vulnerabilities. CyberGym differs through substantially greater scale and diversity, security-focused repository reasoning, and analysis that discovers previously unknown vulnerabilities.

  • Cybersecurity Benchmarks for AI Agents: Existing cybersecurity benchmarks include capture-the-flag problems and benchmarks based on vulnerabilities from real software projects.The paper compares these categories in Table 1.
  • Cybersecurity Benchmarks for AI Agents: CTF benchmarks often use idealized settings that fail to capture real-world cybersecurity complexities.The field has increasingly shifted toward benchmarks based on real-world projects.
  • Cybersecurity Benchmarks for AI Agents: 1,507 instances make CyberGym over seven times larger than any other cybersecurity benchmark, spanning 188 software projects and diverse application domains.The paper links this breadth to a range of task difficulties and gradually improved frontier-model performance.
  • Cybersecurity Benchmarks for AI Agents: CyberGym analyzes agents’ ability to discover zero-day vulnerabilities, whereas other benchmarks focus solely on known historical vulnerabilities.The paper presents this capability as producing direct real-world security impact.
  • Coding Benchmarks for AI Agents: CyberGym focuses on security rather than general software functionality and requires repository-wide reasoning to construct vulnerability-triggering proof-of-concept inputs.This contrasts with SWE-bench and SWT-bench, which often involve localized code changes or testing pull requests.

3 CYBERGYM BENCHMARK

CyberGym is a large, execution-based benchmark built from historical vulnerabilities in diverse C/C++ projects, with tasks and inputs that vary in difficulty. Its design combines reproducible pre-/post-patch validation, sanitizer-based metrics, quality controls, and broad real-world coverage.

  • Scope, Rationale, and Limitations: CyberGym focuses on memory-safety vulnerabilities in widely distributed C/C++ projects detectable through sanitizers, limiting its coverage of broader vulnerability landscapes.The scope is motivated by the frequency and severity of memory-safety issues and by the availability of sanitizers and OSS-Fuzz data.
  • Task Input and Output: Agents receive a vulnerability description and pre-patch codebase, then iteratively create a PoC test in a containerized environment to reproduce the vulnerability.The environment provides execution feedback, including exit codes and command-line output, while agents refine their PoCs.
  • Execution-Based Evaluation Metrics: A PoC succeeds only when it triggers a sanitizer crash before patching and produces no sanitizer crash after patching; the benchmark metric is success rate.This execution-based criterion distinguishes reproduction of the target vulnerability from behavior that persists after the fix.
  • Benchmark Construction: CyberGym’s construction pipeline identifies exact patch commits from OSS-Fuzz histories, assembles pre-/post-patch codebases and PoCs, and applies automated and manual quality filters.Quality assurance includes informative-description checks, reproducibility validation, and removal of redundant or ambiguous instances; human verification on 300 instances found 96% precision.
  • Benchmark Scale and Diversity: 1,507 vulnerabilities across 188 software projects form CyberGym’s benchmark, spanning networks, cryptography, programming tools, scientific computing, operating systems, and multimedia.The dataset covers vulnerabilities disclosed from January 1, 2017, through April 21, 2025, and 62.4% of instances come from projects outside the top 10.
  • Benchmark Scale and Diversity: 28 distinct sanitizer crash types, including buffer overflows and null pointer dereferences, capture diverse vulnerability behaviors.The benchmark’s project and executable diversity includes multiple submodules with differing code and functionality.

4 EXPERIMENTAL EVALUATION

CyberGym presents a demanding evaluation of agents, LLMs, and agent frameworks, differentiating cybersecurity capabilities across task difficulty, reasoning modes, and PoC complexity. Current systems perform best on simpler tasks, while richer inputs improve reproduction and complex tasks remain difficult.

  • CyberGym provides a difficulty ladder that differentiates the cybersecurity capabilities of four agent frameworks and eleven frontier LLMs.
  • 17.9% is Claude-Sonnet-4’s success rate with OpenHands, while specialized software-engineering models achieve success rates ≤2.0%.These results indicate poor generalization from SWE-bench to CyberGym.
  • 22.0% is GPT-5’s success rate with thinking enabled, compared with 7.7% using minimal reasoning.Thinking improves Claude-Sonnet-4 only slightly but substantially increases GPT-5’s success rate.
  • 18.4% is the combined success rate when outcomes across four agent frameworks are united, nearly doubling the best individual result.The low overlap among successful tasks indicates complementary agent capabilities.
  • Around 10% success is achieved on instances with ground-truth PoCs longer than 100 bytes, which comprise 65.7% of the benchmark.Agents also fail more often near the upper limit of 80–100 execution steps.
  • Agents can build executables, perform dynamic testing, and write Python or Bash scripts, but often exhaust iteration limits, request user information prematurely, or overwhelm context windows.

5 FROM BENCHMARKING TO DIRECT SECURITY IMPACT

CyberGym-generated PoCs produced direct security findings beyond benchmark scoring. Validation uncovered zero-day vulnerabilities and incomplete patches in current software versions, while open-ended discovery at scale confirmed additional zero-days.

  • 34 zero-day vulnerabilities were identified and confirmed through CyberGym’s benchmark and open-ended discovery experiments.All were responsibly disclosed; four received CVE assignments and ten had been patched at the reported time.
  • 35 of 759 generated PoCs still caused crashes on latest program versions, yielding 9 unique previously unreported zero-day vulnerabilities after manual analysis and deduplication.These vulnerabilities had existed for an average of 969 days.
  • 18 incomplete patches were confirmed across 15 projects by comparing sanitizer reports and manually checking whether post-patch crashes shared the original root cause.
  • GPT-4.1 triggered 16 crashes and GPT-5 triggered 56 in open-ended discovery across 431 projects and 1,748 entry executables.Manual confirmation identified 7 unique zero-days for GPT-4.1 and 22 for GPT-5, with 4 overlapping.

6 CONCLUSION AND FUTURE WORK

CyberGym is a realistic, large-scale benchmark for tracking AI agents’ cybersecurity capabilities and assessing their real-world security impact. The paper concludes that current systems remain challenged by complex tasks, while future work should broaden the benchmark and improve agent reasoning and coordination.

  • CyberGym contains 1,507 diverse instances across 188 open-source projects and evaluates four agent frameworks and eleven LLMs.Its difficulty ladder is intended to support tracking current and future agent progress.
  • 22.0% is the highest reported success rate for an agent–model combination, while CyberGym also uncovered 34 zero-day vulnerabilities and incomplete security patches.
  • CyberGym currently focuses primarily on C/C++ projects and memory-safety vulnerabilities detected with sanitizers.Future development is proposed for logic flaws, cryptographic weaknesses, web and mobile platforms, and additional programming languages and security tasks.
  • Current agents primarily succeed on tasks with short ground-truth PoCs and fewer reasoning steps, motivating stronger long-context reasoning and ensemble frameworks.The authors also propose specialized security tools and improved tool-use strategies.

ETHICS STATEMENT

Cybersecurity agents have dual-use potential, so CyberGym is framed as a controlled research and evaluation platform requiring responsible design and usage. The benchmark uses patched public data while acknowledging that experiments can reveal previously unknown vulnerabilities.

  • Cybersecurity agents may support protective or offensive applications, making responsible design and usage necessary for CyberGym research.
  • All benchmark vulnerabilities come from public repositories and were patched at least three months before inclusion, reducing immediate ecosystem risk.
  • CyberGym evaluates vulnerability reproduction and discovery in a controlled, reproducible manner based on the established role of fuzzing in vulnerability detection.
  • The benchmark is intended to reveal current limitations and inform development of aligned, controllable, and security-aware agents.
  • The authors position CyberGym as a foundation for transparent research rather than malicious behavior and call for collaboration among researchers, industry, and policymakers.

REPRODUCIBILITY STATEMENT

The paper directs readers to detailed dataset-construction and experimental-setting descriptions, and provides prompts, model checkpoints, agent commits, data, and code to support transparency and reproducibility.

  • Dataset construction is described in Section 3.3, while experimental settings are described in Section 4.
  • Appendix C provides additional details, including prompts, model checkpoints, and Git commits of the agent repositories.
  • The authors open-source the data and code to encourage transparency and reproducibility.The dataset and codebase are linked in the paper.

A THE USE OF LARGE LANGUAGE MODELS

LLMs supported writing, revision, text organization, and data construction, with author review and validation applied to generated or revised text and data.

  • LLMs supported writing, revision, clarity, grammar, style, and organization of written content.
  • LLMs also aided the data-construction process described in Section 3.3.
  • The authors reviewed and validated all text and data generated or revised with LLM assistance.The review was intended to ensure accuracy, appropriateness, and compliance.

B DETAILS OF CYBERGYM BENCHMARK

CyberGym’s benchmark details are organized around general statistics, crash types, and included projects, with project metadata and sanitizer caveats documented in the accompanying tables.

  • Table 3 reports statistics for CyberGym’s benchmark instances.
  • Table 4 lists all crash types and the corresponding numbers of benchmark instances.Most crashes are due to memory-safety issues, but sanitizer-reported crash types may not fully reflect vulnerability root causes.
  • Table 5 lists CyberGym projects together with homepages, languages, GitHub stars, lines of code, and benchmark-instance counts.Most included projects are written in C/C++.

C DETAILS ON EXPERIMENTAL SETUP

The experimental setup combines LLM-assisted benchmark construction, expert auditing, agent PoC generation, configurable model and agent settings, and a new-vulnerability discovery setting built with OSS-Fuzz infrastructure.

  • Benchmark construction: GPT-4.1 filters commit messages and rephrases patch messages into vulnerability descriptions during benchmark construction.Filtering excludes messages lacking informative vulnerability descriptions or addressing multiple issues.
  • Benchmark construction: 300 stratified samples across 96 projects and all crash types are expert-audited to assess LLM-based filtering and rephrasing.The audited subset contains 150 retained and 150 filtered-out samples, with PoC-length distributions similar to the full dataset.
  • Agent evaluation: Agents receive vulnerability-describing files and must generate a single raw input file as a PoC that triggers or exploits the vulnerable program.The workspace provides the necessary information and inputs; CTF agents also receive a flag when a submitted PoC triggers a crash.
  • Agent evaluation: Cybench, EnIGMA, OpenHands, and Codex are evaluated under bounded token, iteration, or cost configurations.Examples include maximum 100 iterations for Cybench, OpenHands, and Codex, and a $2.0 budget for EnIGMA.
  • Model evaluation: Model comparisons vary thinking modes, reasoning effort, output-token limits, and tool use across Qwen3-235B-A22B, Claude-3.7-Sonnet, Claude-Sonnet-4, and o4-mini.The settings include medium reasoning effort for o4-mini and increased output limits for selected thinking-mode configurations.
  • New vulnerability discovery: For new-vulnerability discovery, the benchmark builds latest-available project versions with libFuzzer and AddressSanitizer and uses level 0 for agent-generated PoCs.The setting is described as similar to a fuzzing workflow.

D ADDITIONAL EXPERIMENTAL RESULTS

Additional analyses show that benchmark conclusions remain stable under balanced sampling, while agent behavior reveals distinct command-use patterns, recurrent failure modes, and limited gains from more execution steps.

  • Data contamination: PoC-length distributions were highly similar between pre- and post-knowledge-cutoff splits, indicating no evident contamination confound.The analysis covered GPT-4.1/o4-mini, GPT-5, and Claude-3.7-Sonnet with sufficiently large post-cutoff samples.
  • Robustness analyses: Balanced resampling by software project and crash type produced no significant changes to the conclusions.For OpenHands with Claude-Sonnet-4, success rates across the top crash types ranged from 10% to 25%, compared with an overall rate of 18%.
  • Agent behavior: File searching and browsing dominate agent commands, while CTF-focused agents rely more on task-specific scripts and python3.OpenHands and Codex frequently use ls, whereas EnIGMA primarily invokes framework helper scripts and CTF agents show greater reliance on advanced scripting.
  • Failure modes: Roughly 30% of cases end with premature termination or incorrect success declarations, and approximately 20% involve oversized plaintext PoCs that trigger tool-parsing errors.These failure modes occur alongside repeated PoC attempts, confirmation requests, and overly long command outputs that can exhaust the context window.
  • Implications for tool development: The analysis recommends semantic search, domain-specific tools, scripting, error handling, critical self-evaluation, and stronger backbone-model reasoning.These directions target retrieval inefficiency, repeated low-level operations, premature termination, and incorrect success declarations.
  • Execution budget: Higher execution-step limits yield only marginal improvement: successful runs cluster around steps 20–80, while nearly half of runs fail near steps 80–100.The reported 100-step limit is described as balancing completion of most solvable problems against resource use on intractable cases.

E ANALYSIS OF INCOMPLETE PATCHES AND ZERO-DAY VULNERABILITIES

Agents identified both an incomplete historical patch in GDAL and previously unknown vulnerabilities spanning memory-safety, pointer, allocation, and stack-related failure modes.

  • Incomplete patches: A GDAL patch was incomplete because agents reproduced the null pointer dereference using only the first commit.The vulnerability involved corrupted JPEG streams processed through libjpeg, where required error-handling function pointers were not fully supplied.
  • Zero-day vulnerabilities: The discovered zero-days included 6 out-of-bounds reads, 1 out-of-bounds write, 13 null pointer dereferences, 2 wild-address reads, 1 double free, and 11 stack overflows.The reported root patterns included insufficient error handling, missing boundary checks, and excessive recursion or deep nesting.

F ADDITIONAL CASE STUDIES

The case studies show agents using repository inspection, scripting, and test-case mutation, but often failing to identify or reproduce the precise vulnerability-triggering conditions.

  • Successful reproduction: OpenHands + GPT-4.1 succeeds by copying an existing GIF test case and mutating it with Python scripts.The agent ultimately triggers the target vulnerability after iterative test-case modification.
  • Successful reproduction: GPT-4.1 successfully mutates an existing JPEG test case by adjusting its start-of-scan field to trigger the target vulnerability.Other agents fail through incorrect PoCs, wrong file paths, or malformed test cases.
  • Overall assessment: The agents show promise in file inspection, multi-step reasoning, and scripting, but more sophisticated PoCs require deeper code analysis and better reasoning or frameworks.The paper identifies deeper code comprehension and stronger tools as needed for complex security tasks.
  • Repository exploration: Agents may inspect many files and fuzzer-related directories yet still fail to identify the relevant target or input format.One trajectory searches Wasmtime generator files extensively, while another explores repository files and fuzzer-related build instructions.
  • Failure modes: Some trajectories stop after repeated unsuccessful attempts, including cases where agents fail to trigger a crash or generate long outputs that cause errors.The examples include early termination, repeated unsuccessful PoC generation, and error-producing plaintext output.
Loading 2506.02548v3…