Source-linked AI summary
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, Xiangyu Qi, Eric Wallace, Elie Bursztein, Luca Invernizzi, Kurt Thomas, Yan Shoshitaishvili, Wenbo Guo, Jingxuan He, Thorsten Holz, Dawn Song
TL;DR
AI-agent exploitation remains under-evaluated despite its importance, difficulty, and dual-use risks. ExploitGym addresses this gap with a reproducible benchmark of realistic vulnerabilities, configurable defenses, and validated exploit tasks, showing that frontier agents already exploit a non-trivial fraction of targets while remaining unreliable across the full benchmark.
Problem
Exploitation is an under-evaluated cybersecurity capability requiring low-level runtime reasoning, adaptive progress, and mitigation bypasses, while also carrying defensive and offensive implications.
Method
ExploitGym evaluates agents by having them extend proof-of-vulnerability inputs into validated unauthorized-code-execution exploits across realistic targets, domains, defenses, and reproducible environments.
Results
157 instances were solved by Claude Mythos Preview with Claude Code and 120 by GPT-5.5 with Codex CLI, while defenses reduced but did not eliminate success.
Takeaways & Limitations
Autonomous exploit development by frontier AI agents is no longer hypothetical, although current agents are not yet reliable across all targets.
Takeaways & Limitations
The benchmark excludes several exploitation targets, uses arbitrary code execution as its success criterion, and may conflate exploit-crafting difficulty with refusal, tool misuse, or non-exploitable vulnerabilities.
Abstract
from arXiv · showhide
AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google's V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance. All configurations are packaged in reproducible containerized environments. Our evaluation shows that while exploitation remains challenging, frontier models can successfully exploit a non-trivial fraction of vulnerabilities. For example, the strongest configurations are Anthropic's latest model Claude Mythos Preview and OpenAI's GPT-5.5, which produce working exploits for 157 and 120 instances, respectively. Notably, even with widely used defenses enabled, models retain non-trivial success rates. These results establish ExploitGym as an effective testbed for exploitation and highlight the growing cybersecurity risks posed by increasingly capable AI agents.
1 Introduction
ExploitGym addresses the under-evaluated challenge of turning vulnerabilities into concrete security impacts through low-level, adaptive exploitation. It evaluates agents on realistic vulnerabilities, defenses, and environments, finding that frontier agents already achieve non-trivial exploitation success.
- Exploitation: A Critical Missing Piece in Cybersecurity Evaluation: Exploitation turns an initial vulnerability into stronger primitives and ultimately concrete impacts such as unauthorized file access or code execution.The task requires runtime reasoning about memory layouts, instruction-level control flow, tight input constraints, primitive chaining, and mitigation bypasses.
- ExploitGym: The First Comprehensive Exploitation Benchmark for AI Agents: ExploitGym tasks agents with transforming a proof-of-vulnerability input into a working exploit that achieves unauthorized code execution.Success is validated using a dynamically generated privileged flag inaccessible without unauthorized code execution.
- ExploitGym is Large-Scale, Diverse, and Realistic: 898 instances span 520 userspace tasks, 185 V8 tasks, and 193 Linux kernel tasks derived from real-world vulnerabilities.Each instance is evaluated with and without standard defenses, and all configurations are packaged in reproducible containers.
- Experimental Results Reveal Non-Trivial Exploitation Capabilities: 157 instances were solved by Claude Mythos Preview with Claude Code, compared with 120 solved by GPT-5.5 with Codex CLI.These were the best-performing configurations within a two-hour time limit.
- Experimental Results Reveal Non-Trivial Exploitation Capabilities: Enabling standard defenses substantially reduces success rates but does not eliminate frontier-agent exploitation.The results motivate stronger exploit-resistant defenses and responsible model development and deployment.
2 Related Work
ExploitGym builds on vulnerability-discovery infrastructure and prior cybersecurity benchmarks while converting existing artifacts into standardized exploitation evaluations. Compared with earlier benchmarks, it targets realistic, complex exploitation across broader software domains and configurable defenses.
- Vulnerability Discovery and Exploitation: OSS-Fuzz, syzbot, kernelCTF, and related systems provide vulnerability artifacts but were not designed as standardized benchmarks.ExploitGym transforms these artifacts into controlled tasks with standardized prompts, execution environments, configurable defenses, and reliable exploit validation.
- Vulnerability Discovery and Exploitation: Prior automated exploit-generation approaches often rely on strong assumptions about vulnerability classes.Existing techniques include stack-smashing attacks, return-oriented programming, and data-oriented programming designed to bypass mitigations.
- Cybersecurity Benchmarks for AI Agents: Existing exploitation benchmarks such as NYU CTF, Cybench, CVE-Bench, and BountyBench contain at most 200 instances and primarily use synthetic or userspace settings.ExploitGym is compared against these benchmarks across capture-the-flag, userspace, browser, and kernel domains.
- Cybersecurity Benchmarks for AI Agents: Higher ExploitGym scores indicate stronger exploitation abilities or risks, but not necessarily better or safer models overall.The benchmark therefore evaluates a potentially dangerous capability rather than general model quality or safety.
3 ExploitGym Benchmark
ExploitGym evaluates agents by turning vulnerability-triggering inputs into working exploits across three software-stack domains and configurable mitigation settings. Its protocol combines controlled remote interaction with flag capture and trajectory-based verification, using reproducible artifacts sourced from real-world vulnerabilities.
- 3.1 Evaluation Protocol: Each task pairs a vulnerability with a mitigation setting and supplies build, vulnerability, and runtime information for agent interaction.Agents interact with a restricted remote target and can use multiple rounds plus resets to a clean state.
- 3.1 Evaluation Protocol: Success requires both capturing a protected flag through unauthorized code execution and passing an agent-as-a-judge assessment of vulnerability targeting.The judge checks that the agent did not succeed through an unrelated vulnerability, shortcut, or known public exploit.
- 3.2 Task Domains: ExploitGym spans userspace programs, the V8 JavaScript engine, and the Linux kernel, with domain-specific security models and independently toggled defenses.Userspace tasks target memory-safety vulnerabilities, V8 tasks include OS-level ASLR and the heap sandbox, and kernel tasks require privilege escalation across a sandbox boundary.
- 3.2 Task Domains: Table 2 organizes the benchmark by exploitation surface and lists mitigations as independent toggles.This layout supports comparisons across distinct software layers and defense configurations.
- 3.3 Benchmark Construction: Benchmark construction sources vulnerabilities from OSS-Fuzz, Chromium issue data, kernelCTF, and syzbot, then reproduces and validates them in controlled environments.The pipeline includes PoV recovery or generation, vulnerable-build verification, deduplication, and curation of agent-facing artifacts.
- 3.3 Benchmark Construction: Table 3 reports intended-vulnerability successes by total and domain, alongside estimated cost and per-task averages over successful and full benchmark subsets.The experiments use a two-hour timeout and trusted-access programs with safeguards disabled.
4 Evaluation
ExploitGym evaluates frontier agents across diverse real-world vulnerabilities, defenses, domains, time budgets, and validation criteria. Results show substantial but uneven exploitation ability, with complementary model strengths, sustained long-horizon gains, and meaningful resilience to mitigations.
- Overall results: 157 and 120 successes were achieved by Claude Mythos Preview and GPT-5.5, respectively, under a two-hour timeout with mitigations disabled.These were the highest success counts among evaluated configurations; GPT-5.4 solved 54 tasks, while remaining pairings solved fewer than 15 each.
- Safety filters: 88.2% of GPT-5.5 runs with default safety filters were blocked before any tool call, while the remainder stayed in reconnaissance without progressing toward exploitation.The unblocked runs averaged 4.4 valid LLM requests but made no exploitation progress.
- Evaluation validity: Flag-to-success alignment ranged from 36.4% for GLM-5.1 to 83.1% for GPT-5.4, showing that code execution often used vulnerabilities other than the intended target.GPT-5.5 and Claude Mythos Preview aligned at 56.7% and 69.5%, respectively, with 90 and 69 unintended-path solves.
- Domain differences: Kernel tasks most sharply separated models: Claude Mythos Preview and GPT-5.5 achieved 12 and 22 successes, while no other model exceeded one.Userspace tasks were broadly successful, whereas V8 was substantially harder and successes were concentrated among a few models.
- Time scaling: Claude Mythos Preview continued accumulating successes beyond two hours without a clear plateau, unlike Claude Opus 4.6’s rapid saturation.The trajectory is consistent with sustained multi-stage exploit refinement and suggests that a two-hour budget under-counts Mythos Preview’s capability.
- Complementary capabilities: 56 targets were solved exclusively by Claude Mythos Preview and 26 exclusively by GPT-5.5, with 91 shared successes.The remaining models contributed 61 successful exploits, including four uniquely solved by them, indicating that unions of agents can expand coverage.
- Mitigations: Standard mitigations eliminated the majority of agent-generated exploits, yet some agents bypassed ASLR, V8 sandboxes, and KASLR through concrete exploit strategies.Examples include partial-pointer overwrites, low-bit brute force, Wasm or Irregexp primitives, writable kernel strings, and side-channel leaks.
- Case study: Figure 4 illustrates a V8 trajectory from an assertion-triggering PoV through an out-of-bounds read, arbitrary native memory reads, dispatch hijacking, and privileged flag access.An independent scorer verifies whether the exploit targets the specified vulnerability.
5 Discussion and Conclusion
The benchmark has important scope and measurement limits, but its results show that frontier agents can already autonomously exploit a non-trivial fraction of realistic vulnerabilities. This emergence motivates evaluating defenses against AI-driven attackers and developing responsible safeguards.
- The benchmark excludes major target environments, including Windows, iOS, Android, and applications running in those environments.
- Arbitrary code execution is the success criterion, excluding outcomes such as arbitrary read/write primitives, sandbox escape without code execution, and partial exploit progress.
- Failures can reflect safety refusals, tool misuse, non-exploitable vulnerabilities, or the absence of ground-truth exploits rather than payload-crafting difficulty alone.
- Exploit generation is dual-use: defenders can assess vulnerability severity, while attackers may reduce development costs and barriers to exploitation.
- ExploitGym shows that autonomous exploit development is no longer hypothetical, although current agents remain unreliable across all targets.
- Responsible model development and stronger defenses should incorporate autonomous exploitation into threat modeling.
A Ethics and Impact Statement
AI-agent exploit generation has defensive value but also creates offensive risks because the capability is inherently dual-use. ExploitGym is positioned as a controlled evaluation resource for understanding these risks and informing defensive investment.
- The same exploit-generation techniques can help defenders assess risks while lowering the expertise barrier for offensive misuse.
- The benchmark uses publicly available sources and includes vulnerabilities patched upstream before inclusion, reducing immediate risk to the software ecosystem.
- Exploit development supports severity assessment, mitigation validation, and defense-in-depth strategies.
- ExploitGym enables realistic evaluation of agent capabilities and helps inform AI safety evaluations and defensive investments.
B Benchmark Details
This appendix section provides additional details about how benchmark data was collected and filtered.
- The appendix documents the data-collection and filtering pipelines for each benchmark category.
B.1 Userspace Software
Userspace instances are drawn from reproducible vulnerability corpora and prepared for deeper exploitation under varied mitigation settings. The construction excludes trivially exploitable interpreter targets and disables sanitizers that would otherwise halt execution at the first memory violation.
- CyberGym Instances: CyberGym userspace instances are memory-safety vulnerabilities in C/C++ projects with reproducer inputs, upstream patches, and reproducible Docker environments.
- CyberGym Instances: Targets whose fuzzing entry points already expose shell or filesystem APIs are excluded because code execution would be trivial without exploiting memory corruption.
- CyberGym Instances: Each target is rebuilt without sanitizers because sanitizer aborts would prevent deeper exploitation after the initial memory violation.
- OSV Instances: OSV instances supplement fuzzer-discovered bugs with vulnerabilities found through other methods, such as code audits.
- Mitigation Configuration: The benchmark varies ASLR, stack canaries, and position-independent code to control mitigation configurations.
B.2 Browser (V8)
The browser instances are drawn from post-2024 Chromium reports and use recovered or attached proof-of-vulnerability inputs. V8-specific shell reductions and configurable mitigations prevent trivial flag retrieval and enable controlled evaluation.
- ClusterFuzz Instances: ClusterFuzz instances target post-2024 Chromium reports on V8, recovering inputs from unit tests shipped alongside patch commits.The post-2024 restriction makes the V8 heap-sandbox mitigation toggle meaningful.
- Human-Reported Instances: Human-reported instances retain post-2024 reports with attachments plausibly serving as proof-of-vulnerability inputs.Candidate vulnerable commits are identified from report text and validated by executing the attached proof of vulnerability.
- Shell Surface Reduction: The patched d8 shell disables convenience APIs that could provide trivial flag retrieval, retaining only APIs needed by real-world proof-of-vulnerability inputs.The retained APIs are setTimeout, Worker, and d8.serializer.
- Mitigation Configuration: The browser experiments independently toggle ASLR at the operating-system level and the V8 heap sandbox at build time.The heap-sandbox setting is controlled with the v8_enable_sandbox flag.
B.3 Linux Kernel
The Linux-kernel instances combine kernelCTF submissions with selected syzbot reports, using simplified vulnerability-triggering inputs. Privilege controls and configurable kernel mitigations require exploitation to cross the intended sandbox boundary.
- kernelCTF Instances: kernelCTF entries provide target kernels, source and build metadata, root filesystems, ground-truth exploits, and detailed vulnerability materials.A human–agent collaboration pipeline simplifies each full exploit into a minimal proof of vulnerability that triggers the bug.
- syzbot Instances: syzbot instances target x86 and x86_64 high-severity memory-safety and data-race bugs with reproducible kernel artifacts and commits.For duplicate underlying bugs, upstream reports and the earliest timestamp are preferred.
- Privilege Control: Agents run under nsjail restrictions, and obtaining UID 0 only inside a user namespace cannot access the flag device.Success therefore requires privilege escalation beyond the sandbox boundary.
- Mitigation Configuration: The kernel experiments toggle KASLR and user-namespace creation through the nokaslr boot parameter and kernel.unprivileged_userns_clone sysctl.User namespaces are treated as an additional attack surface when permitted.
C Experiment Details
The experiments evaluate six models with officially recommended coding agents under documented reasoning settings. Runs use specified hardware, network, and container controls to standardize execution and limit external access.
- Models and Agents: Six models are accessed through official APIs and paired with their providers’ officially recommended coding agents.Reasoning effort is set to the highest available level where supported, except for Claude Opus 4.7 because early termination is frequently observed.
- Models and Agents: Table 6 summarizes the evaluated model checkpoints, agent versions, and reasoning-effort settings.All models in the table are accessed through official API endpoints.
- Experiment Environment: Experiments use GCP c4-standard-96 and c4-standard-288 instances plus a dual-AMD-EPYC system with specified CPU and memory capacities.The listed environments range from 96 to 288 vCPUs and from 384 GB to 1,080 GB of RAM.
- Network Restrictions for Agents: Agent network access is mediated by an egress proxy that permits Docker networking, selected package repositories, and required V8 toolchains while blocking other external endpoints.The restrictions reduce security risks and potential reward hacking through web search.
- Resource Isolation: Each agent runs in a Docker container limited to 4 CPU cores and 8 GB of memory.These limits standardize resources across runs and prevent one agent from monopolizing host capacity.