CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

• Existing benchmarks often omit remediation or evaluate stages separately, while others have limited scale, synthetic data, unrealistic environments, or insufficient post-patch testing.Source: §1 · p. 1
• CyberGym-E2E is a realistic, large-scale end-to-end benchmark built from real-world vulnerability data, with a construction method designed for realism, diversity, and scale.Source: §1 · p. 1
• Its pipeline uses real-world repositories, ground-truth fixing commits, developer-written unit tests, and expert review to generate end-to-end tasks with reduced manual overhead.Source: §1 · pp. 1–2
• Agents achieve high security-patch success, but vulnerability detection and PoC generation remain challenging and lower end-to-end performance; the benchmark covers 920 vulnerabilities across 139 projects.Source: §1 · p. 2
• The authors describe CyberGym-E2E as the first scalable methodology for end-to-end security benchmark construction and the first large-scale evaluation across the vulnerability lifecycle.Source: §1 · p. 2

2. Background and Related Work

• A vulnerability typically proceeds through discovery, PoC generation to reproduce it, and remediation through a patch.Source: §2 · p. 2

2.1. Comparing CyberGym-E2E to Other Benchmarks

• BountyBench evaluates end-to-end vulnerabilities but covers only 40 manually curated tasks; SeCodePLT and SEC-bench scale more broadly but omit full-lifecycle evaluation or have limited post-patch testing.Source: §2.1 · p. 2
• The paper groups benchmark shortcomings around incomplete lifecycle coverage, unrealistic environments, and insufficient functionality testing or scale.Source: §2.1 · p. 2
• Many benchmarks isolate lifecycle stages: some focus on offensive discovery and PoCs, others on remediation, while SeCodePLT and SEC-bench include separate rather than consecutive tasks.Source: §2.1 · p. 2
• CyberGym-E2E runs agents in the project build environment with sandboxing, unlike benchmarks that often restrict access to read-only codebases or limited functions.Source: §2.1 · pp. 2–3
• Existing patch evaluations may lack functionality tests or rely on limited fuzzing or state-difference checks; SecureAgentBench and SecRepoBench use developer-written tests but lack scale.Source: §2.1 · p. 3

3.1. Preliminaries

• Fuzzers and sanitizers help identify reproducible vulnerabilities in OSS-Fuzz projects; ARVO packages some vulnerabilities but lacks evaluation tasks and functional tests and does not regularly ingest new data.Source: §3.1 · p. 3

3.2. Overview

• The design aims for realistic, reproducible, scalable, end-to-end evaluation, including Dockerized environments and automation; legacy vulnerable projects are migrated to newer operating systems while preserving PoC behavior.Source: §3.2 · pp. 3–4
• Developer-written unit tests are extracted and checked for coverage around vulnerable code; an agent-assisted pipeline prepares them, and an expert manually validates correctness and coverage.Source: §3.2 · p. 4
• The dataset contains 920 vulnerabilities across 139 projects; repositories have a median of 1,811 files and 613,227 lines of code, while patch sizes range from 1 file and 7 lines to 29 files and 3,988 lines.Source: §3.2 · p. 4, Table 2 · p. 4

3.3. Benchmark Construction Pipeline

• The pipeline locates a clean fixing commit through repository history, then checks vulnerable and patched builds and confirms that the ground-truth PoC triggers only on the vulnerable version.Source: §3.3 · p. 4
• An agent identifies, builds, and runs developer test suites in a Dockerized patched project; an expert checks harness behavior and coverage, rejecting projects with inadequate or failing tests.Source: §3.3 · p. 5
• Filtering reduced roughly 1,200 candidates to about 800 at test preparation, then expert review accepted 74% and rejected 26%, producing 615 initial tasks before later expansion to 920.Source: §3.3 · p. 5

3.4. Task Format and Evaluation

• Patch-only evaluation supplies ground-truth PoCs and crash logs, while end-to-end evaluation withholds them; validation checks agent crashes, patch fixes, preserved tests, and whether the original PoC is also fixed.Source: §3.4 · p. 5, Figure 2 · p. 5

4.2. Main Results

• 82.3% patch-only success for Opus 4.5 with Claude Code falls to 19.2% on end-to-end S3, making vulnerability discovery the primary bottleneck.Patch-only agents receive the ground-truth PoC and crash log; S3 requires passing earlier lifecycle stages and passing tests on the patched version.Source: §4.2 · p. 6, Table 3 · p. 6
• Opus 4.5 leads patch-only performance, while GPT-5.2-Codex and Gemini 3 Pro outperform it in detection; over half of Opus runs hit the cost cap.Opus’s lower S1 performance keeps its S3 score below GPT-5.2-Codex and Gemini 3 Pro, despite stronger patching capability.Source: §4.2 · p. 6
• S3 exceeds S4 when agents fix a different vulnerability in the same code region; some successful patches instead add shallow guards that leave the defect untouched.S4 requires eliminating the crash from the ground-truth PoC, so it checks whether the agent found the same vulnerability.Source: §4.2 · pp. 6–7

4.3. Harness Comparison

• OpenHands’s whole-file reading consumes more tokens, while targeted search and Claude Code’s task tracking support more systematic exploration.OpenHands trajectories are more expensive and achieve lower success rates; targeted approaches enable deeper exploration within the same budget.Source: §4.3 · p. 7

4.4. Ablation Studies

• Longer time and higher cost budgets yield diminishing returns; Opus 4.5 rises from 11.0% to 19.2% S3 success between $5 and $10.The $10 cap is an evaluation parameter, not a constraint on the benchmark dataset.Source: §4.4 · p. 7
• Cross-run feedback improves success rates by 5-7 percentage points; in S3, pre- and post-model-knowledge-cutoff performance is statistically indistinguishable, with both Fisher’s exact test and Z-test p-values above 0.1.The cutoff analysis uses the initial 615 tasks and some subsequently curated tasks with disclosure dates after 2025, and excludes Gemini 3 Pro.Source: §4.4 · p. 8

4.5. Agent Behavior Analysis

• Successful vulnerability-discovery trajectories parse descriptions, search targeted code, trace triggering conditions, construct PoCs, and refine them using validation feedback; failures stem from analysis gaps, resource exhaustion, or ineffective exploration.Source: §4.5 · p. 8
• The testing harness blocks edits to evaluation-related files, yet trajectories show unverified success claims and selective reporting, motivating rigorous non-agent-dependent evaluation.The authors also call for anticipating metric optimization, sanitizing data-leakage sources, and establishing clear rules of engagement.Source: §4.5 · p. 8

5. Limitations

• The benchmark focuses on memory-safety vulnerabilities in C/C++ and uses sanitizer-triggered crashes to validate PoCs and patches.The choice reflects the prevalence of memory-safety bugs, sanitizers’ reliability as automated oracles, and OSS-Fuzz’s structured vulnerability data.Source: §5 · pp. 8–9
• Logic bugs, injection vulnerabilities, concurrency bugs, and web security issues may not trigger sanitizer crashes and require different evaluation oracles.The framework is oracle-agnostic and could be extended with other programmatic success/failure harnesses.Source: §5 · p. 9
• Included projects primarily process structured inputs, which may constrain software diversity.An example trajectory follows an agent locating a GraphicsMagick heap-buffer-overflow, constructing an MNG PoC, and iteratively refining a patch through validation.Source: §5 · p. 9, Figure 3 · p. 9

6. Conclusion and Future Work

• CyberGym-E2E evaluates the vulnerability lifecycle across 920 historical vulnerabilities from 139 open-source projects.The pipeline transforms historical vulnerability data into evaluation environments and test suites.Source: §6 · p. 9
• Future work aims to use code coverage analysis to automate human validation of test coverage and improve correctness testing for agent-provided patches.Possible sources include developer-provided coverage assessments or LLVM and Clang coverage to check whether unit tests cover the ground-truth patch.Source: §6 · p. 9
• Future work aims to expand beyond C/C++ using sources such as CVE records and GitHub vulnerability databases to add languages including Python, Java, Rust, and Go.The proposed expansion targets a broader range of security issues, including injection, deserialization, and access-control weaknesses.Source: §6 · p. 9

Impact Statement

• The authors argue that rising AI capabilities and agent-assisted real-world attacks make evaluation of end-to-end cybersecurity capabilities urgent.CyberGym-E2E is presented as addressing scale, realism, and validation limitations in prior benchmarks.Source: Impact Statement · p. 10
• Vulnerability-discovery capabilities are dual-use: the benchmark supports defensive development, but could potentially lower barriers to offensive cyber activity if misapplied.All included vulnerabilities were publicly disclosed and remediated before inclusion, and the benchmark evaluates patch generation as well as discovery.Source: Impact Statement · p. 10
• The authors consider transparent evaluation of capabilities, including strengths and limitations, essential for developing safeguards and informing responsible cybersecurity deployment.The stated aim is to support defensive systems rather than only accelerate attacks.Source: Impact Statement · p. 10

B. Prompts provided for agents

• The prompts give explicit instructions for iterative workflow and validation in both end-to-end and patch-only evaluations.Figure 4 shows the end-to-end prompt, while Figure 5 shows the patch-only prompt.Source: B. Prompts provided for agents · p. 12
• The end-to-end prompt uses section keywords, file paths, executable commands, and warnings to organize agent instructions.Blueprints are described in Figure 4’s caption as headers, locations, commands, and critical notes.Source: Figure 4 · p. 13
• The patch-only prompt supplies the ground-truth PoC and crash log, isolating the task to root-cause analysis and patch generation.The project table includes homepage links, primary languages, GitHub stars, lines of code, and benchmark-instance counts.Source: Figure 5 · p. 14, Table 9 · p. 15, B. Prompts provided for agents · p. 15