Source-linked AI summary
ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents
Seunghyun Lee, David Brumley
TL;DR
Existing evaluations often treat a crash as exploitation success, leaving unclear how far an agent progresses toward reusable primitives and control. ExploitBench addresses this gap with a 16-flag capability ladder and deterministic verification across 41 hardened V8 bugs and multiple measurement arms. Public frontier models routinely reach vulnerable code and crashes but rarely reach advanced exploitation, whereas Mythos Preview reaches arbitrary code execution on 18 of 41 bugs.
Problem
Existing benchmarks collapse exploitation into binary outcomes, failing to measure the progression from bug triggering to reusable primitives, sandbox escape, and control.
Method
ExploitBench evaluates 16 exploitation capabilities with deterministic challenge-response, differential-execution, and signal-handler oracles across 41 hardened V8 bugs and three measurement arms.
Results
Public frontier models routinely reach vulnerable code and crashes but rarely cross into advanced exploitation, while Mythos Preview reaches arbitrary code execution on 18 of 41 bugs.
Takeaways & Limitations
The results place exploit construction against hardened targets near an emerging frontier boundary, with public models stalling before the primitives needed to escape the V8 heap sandbox.
Takeaways & Limitations
EXPLOITBENCH measures exploit development in a controlled harness, not weaponization or reliability across uncertain builds, shifting heap state, or operational defenses.
Abstract
from arXiv · showhide
Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target. However, existing LLM security benchmarks treat a crash as exploitation success. That single binary outcome collapses the hard parts of exploitation: the transition from triggering a bug to constructing reusable primitives and control. We present ExploitBench, a capability-graded benchmark that decomposes exploitation into 16 measurable flags, from coverage and crash through sandbox primitives, arbitrary read/write, control-flow hijack, and arbitrary code execution. Each capability is verified by a deterministic oracle that uses a per-run randomized challenge-response for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution. We instantiate ExploitBench on 41 V8 bugs because V8 is both widely deployed and exploitation-hardened. We report three arms: <model,env> as the primary measurement of model-environment capability, <model,env, adaptive coaching> as a secondary arm that adds adaptive coaching to test whether targeted feedback shifts outcomes, and <model,env,harness> as an ablation that swaps in the model's native CLI to check whether vendor-side optimizations increase exploitation capabilities. Our results show a sharp capability split between publicly deployed frontier models and the private frontier. Across the 8 publicly deployed models tested, reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not. The private model shows arbitrary code execution on approximately half. Overall, results suggest that exploit construction against hardened targets is an emerging frontier capability.
1 Introduction
ExploitBench addresses the gap between reproducing a vulnerability and demonstrating progressively stronger exploitation capabilities on hardened production software. It combines capability-graded evaluation, deterministic verification, and multiple measurement arms to separate model capability from scaffolding effects.
- Existing benchmarks often collapse exploitation into pass/fail outcomes, obscuring the difference between triggering crashes and constructing reusable primitives or control.
- The paper asks how far agents progress from bug triggering toward weaponization, which vulnerability properties predict advanced capabilities, and how deterministic grading and scaffolding affect results.
- Three arms compare a uniform bare-model setting, adaptive coaching, and native vendor CLI scaffolding to distinguish model-environment capability from harness effects.
- EXPLOITBENCH decomposes exploitation into 16 independently measurable flags spanning coverage, bug detection, engine-specific primitives, general-purpose primitives, and full exploitation.
- Each capability uses deterministic verification, including challenge-response builtins, differential execution, and signal-handler-based checks for control-flow hijack and arbitrary code execution.
- The benchmark evaluates 41 recent V8 bugs in default release builds with deployed mitigations enabled, targeting exploitation of a widely deployed, hardened engine.
- Public frontier models usually reach vulnerable code and crashes, but only one public model reached advanced primitives on one WebAssembly bug, while Mythos Preview reached arbitrary code execution on 18 of 41 bugs.
2 Design
The design packages each known vulnerability into a reproducible environment and grades a monotonic, five-tier capability ladder against untouched binaries. Deterministic differential, primitive, and control-flow oracles make advanced claims testable without relying on human judgment.
- 2.1 Environment: Each environment contains the vulnerable source tree, vulnerable and fixed binaries, applicable capability flags, and a prompt describing the bug, patch, and grader builtins.
- 2.2 Interface: EXPLOITBENCH uses a uniform MCP interface and pinned dependencies to support reproducible cross-model comparisons and reduce environment drift.
- 2.1 Environment: The agent may modify and instrument the source during exploration, but grading runs submissions against untouched ground-truth binaries, preventing local changes from affecting scores.
- 2.3 Capability Ladder: The 16 capability flags form five ordered tiers from reaching code and triggering bugs through engine primitives, general-purpose primitives, and control-flow hijack.
- 2.3 Capability Ladder: Coverage and bug triggering are graded through patch-location coverage and differential execution across vulnerable and fixed builds, with crash and sanitizer outcomes providing distinct signals.
- 2.3 Capability Ladder: Control-flow hijack requires reaching a randomized unknown page, while arbitrary code execution requires attacker-chosen code to perform an attacker-chosen action on the JavaScript thread.
- 2.4 Arms: The benchmark separates adaptive coaching as a secondary arm because coaching changes trajectories differently across models and would otherwise mix instruction-following with capability.
3 Implementation
EXPLOITBENCH implements evaluation as a containerized system whose server, image builder, and runner jointly provide uniform interaction and deterministic grading. Instrumented V8 binaries execute multiple randomized grading rounds, while offline audits check run integrity.
- The implementation has three components: an in-container MCP server, an environment builder that creates Docker images, and an external agent runner.
- The MCP server exposes six tools and a composite grader covering coverage, differential execution, and exploit primitives.
- A C++ V8 extension registers grader functions and installs alternate-stack signal handlers so randomized control-flow and code-execution oracles run inside d8.
- The environment builder retrieves vulnerable and fixed revisions, extracts patch locations, builds multiple V8 configurations, and assembles self-contained images.
- The runner standardizes prompts, tool schemas, budgets, and audit trails across providers while using direct APIs to normalize model evaluation.
- Each grade call uses twelve d8 invocations across randomized rounds and binaries, crediting a capability only when all three rounds agree.
- A deterministic offline audit checks transcript and score artifacts for integrity violations, including protected-path access and hardcoded-address cheats.
4 Evaluation
ExploitBench evaluates 41 V8 bugs across nine frontier models and three measurement arms, using a 300-turn budget and best-of-three capability ceilings. Results show a sharp split: most public models stall below sandbox escape, while Mythos Preview reaches the top of the ladder repeatedly.
- Evaluation setup: 41 V8 bugs were evaluated across nine frontier models, with three seeds and a 300-turn budget per cell; headline scores use the best-of-three union.The primary and coaching arms cover all models, while the vendor-CLI arm is run for GPT-5.5 only.
- RQ1: How far up the pipeline?: Only Mythos Preview reached ace at scale, achieving ace on 18 of 41 bugs in the bare-model primary arm.The other eight agents did not cross beyond Tier 2 in the primary arm, although some reached Tier 3 on multiple bugs.
- RQ1: How far up the pipeline?: Mythos completed successful exploit chains in roughly 110–200 turns, after reaching coverage and engine primitives within about 40 turns and cage-bounded read/write by turn 60.Each successive step took roughly an order of magnitude more turns than the previous one, with most wall-clock time spent on the final two steps.
- RQ2: Where does the boundary lie?: Four agents stalled at T4→T3, while four others stalled at T3→T2; Mythos showed no comparable boundary on cells reaching Tier 2 or higher.The observed stalls occurred at distinct transitions: some agents triggered bugs without building engine primitives, while others built primitives without crossing the sandbox.
- RQ3: What predicts success?: WebAssembly type-confusion bugs accumulated capabilities faster and reached higher tiers than JavaScript-only bugs, although model identity became at least as predictive as bug class.Mythos reached ace on 18 bugs, whereas the next-best agent on any bug reached ace on at most one.
- RQ5: Harness effect across arms: Adaptive coaching raised GPT-5.5’s Tier-3 count from 13 to 22 bugs but reduced Gemini 3.1 Pro’s Tier-3 count from 16 to 8, showing mixed harness sensitivity.For Mythos, coaching increased Tier 3 from 35 to 37 bugs but reduced pc control and ace counts from 18 to 16.
- RQ5: Harness effect across arms: Codex CLI increased GPT-5.5’s Tier-3 coverage from 13 to 20 bugs at roughly one-fifth the per-episode cost, but added only one top-level flag on one bug.The bare model already reached pc control and general primitives on that bug; the CLI closed the remaining ace flag.
5 Discussion
The discussion frames exploitation as a capability boundary rather than a binary outcome, while showing how harness choices, deterministic grading, and scope limitations shape interpretation of results.
- Measurement and reliability: Deterministic oracles make grading reproducible because the same submission and seed produce the same capability bitmap.This supports capability benchmarks as more reliable reward sources than LLM-judged evaluations.
- Measurement and reliability: Three randomization rounds must agree before a flag receives credit, deliberately undercrediting flaky exploits.
- Capability boundary: Publicly deployed models stall at or before engine primitives, whereas the non-public frontier reaches advanced exploitation within the 300-turn budget.The discussion attributes this gap to a capability boundary rather than simply insufficient turns, because staircase costs grow roughly an order of magnitude per capability.
- Harness effects: Adaptive coaching changes models asymmetrically, while vendor CLI scaffolding adds only one flag on one bug for GPT-5.5.GPT-5.5 rises from 13 to 22 Tier-3 bugs under coaching, whereas Gemini 3.1 Pro falls from 16 to 8; Codex extends GPT-5.5 from pc control to ace on one bug.
- Defensive interpretation: The ladder also supports defensive prioritization by distinguishing crashes, intermediate capabilities, and full control as different operational severities.The paper describes rung-level grading as a replacement for binary crash triage and notes that weaponization and reliability remain outside the benchmark.
- Scope and limitations: The 1-day-with-patch framing leaks coverage information into lower-band capabilities, making the coverage column uninformative on its own.The authors retain the full patch-time commit rather than manually removing tests that trigger basic capabilities.
6 Related Work
Prior benchmarks establish automated vulnerability reproduction across web, software, and network settings, but mostly reduce exploitation to binary success. ExploitBench instead compares exploitation ladders, scaffolding, and reproducibility constraints against concurrent work.
- Web and PoC benchmarks: BountyBench and CVE-Bench evaluate web vulnerabilities through detect, exploit, and patch or sandboxed CVE tasks, but do not measure progressive exploitation capabilities.BountyBench covers 40 bug bounties across 25 systems.
- Web and PoC benchmarks: PoC reproduction benchmarks scale across many vulnerabilities but typically award success when an agent triggers a sanitizer or buggy path.The related work contrasts triggering a bug with constructing addrof, fakeobj, arbitrary read/write, ASLR bypass, and code execution.
- Adjacent security evaluations: CTF, network-attack, threat-intelligence, defensive-code, and bug-introduction evaluations address adjacent security tasks rather than offensive exploit generation.
- Concurrent benchmarks: ExploitGym is complementary because it asks what fraction of bugs agents solve, whereas ExploitBench asks where agents stall on the exploitation ladder.ExploitBench also reports bare-model, adaptive-coaching, and vendor-CLI configurations rather than evaluating each model through one vendor CLI.
- Evaluation design: Reproducibility remains a bottleneck in prior frameworks, while ExploitBench uses a protocol-level uniform interface to reduce environment and agent confounds.The benchmark comparison table is presented as a design-axis comparison with concurrent benchmarks.
7 Conclusion
ExploitBench evaluates V8 exploitation as a graded, deterministic progression across 16 flags and three measurement arms. Results show strong public-model performance at early stages but a pronounced boundary before sandbox escape, while harness effects are smaller and model-dependent.
- Benchmark and measurement: ExploitBench evaluates 41 V8 bugs across 16 measurable flags, using primary, adaptive-coaching, and vendor-CLI arms.The benchmark separates model–environment capability from coaching and vendor scaffolding.
- Results: Six of eight publicly deployed models reached Tier 3 in the primary arm, but only GPT-5.5 advanced beyond it.GPT-5.5 reached Tier 2 on two Wasm bugs and PC control on one.
- Results: The public panel’s conditional Tier 3→Tier 2 transition rate was near zero, while Mythos Preview reached ace on 18 of 41 bugs.The comparison indicates a sharp capability boundary between the public panel and private reference under the same primary setup.
- Results: Bug class favored public-model progress, with WebAssembly type-confusion bugs reaching higher tiers than JavaScript-only or JIT-compiler bugs.For Mythos Preview, ace results spanned Wasm, JIT-compiler, and historical-cohort bugs.
- Harness effects: Adaptive coaching changed models in opposite directions, while the vendor CLI added ace on one public-model bug at roughly one-fifth the per-episode cost.The vendor-CLI lift added one flag on one public-model bug, compared with eighteen for the public-versus-private model gap.
- Implications: The benchmark replaces binary crash triage with rung-level grading for assessing bug severity, reproducing vulnerabilities on shipping builds, and prioritizing patches.The authors frame this capability ladder as a dual-use defensive triage signal.
A Open Science
The Open Science appendix requires authors to enumerate evaluation artifacts, explain reviewer access, and justify any artifacts that cannot be shared. When full sharing is impossible, partial, synthetic, or redacted artifacts are encouraged.
- Artifact disclosure: Authors must enumerate all artifacts needed to evaluate the paper’s core contributions.Examples include code, datasets, models, configuration files, scripts, documentation, and benchmarks.
- Reviewer access: Authors must describe how the program committee can access each artifact during double-blind review.Anonymous URLs or credentials should be provided where applicable.
- Sharing constraints: Unshareable artifacts require explicit justification based on licensing, disclosure, safety, privacy, or deployment risks.The appendix encourages partial, synthetic, or redacted artifacts when full sharing is not possible.
- Artifact disclosure: If no artifact is needed to evaluate the core contributions, authors should state that explicitly.
B Ethical Considerations
The paper frames its cybersecurity experiments as dual-use research aimed at measurement rather than operational offense. It uses historical vulnerabilities with patches available and reports no external influence on the measurements or conclusions.
- Dual-use framing: The study acknowledges that asking models to create weaponized exploits is inherently dual-use.The stated purpose is rigorous measurement of LLM capabilities in the wild, rather than offensive deployment.
- Risk mitigation: The experiments use historical vulnerabilities with patches already available and report no confirmed new zero-days.The authors state they confirmed this to the best of their ability.
- Independence: Anthropic provided API credits but had no involvement in measurements, interpretation, analyses, or conclusions.The authors report that all experiments and conclusions were conducted solely by the authors.
C Episode prompt
The episode prompt uses a common system message and renders each task from bug-specific V8 information, including the bug identifier, description, patch diff, and capability table.
- Prompt structure: Every model receives the same system message at episode start.The prompt is standardized across bugs.
- Prompt structure: The task message is rendered per bug from its identifier, description, patch diff, and capability section.
System message
The system message requires agents to inspect the actual patch and source after setup before analyzing or executing against a V8 vulnerability.
- Agents must call setup() and read the actual patch diff and source code before summarizing or speculating about a vulnerability.Prior knowledge about a CVE is explicitly treated as unreliable.
- All execution must use tool calls, with stdout and stderr truncated to 2,000 characters.Command executions and evaluation runs have 60-second timeouts.
- Processes terminated with exit code 137 indicate SIGKILL under the stated execution constraints.
Task message (V8 template)
The V8 template defines the benchmark environment, figures, measurement arms, and capability comparisons used to evaluate exploitation progress across models and bugs.
- Environment: The benchmark uses vulnerable and fixed V8 d8 binaries across debug, ASAN, release, and coverage configurations.The source tree and prebuilt binaries are available for rebuilding and differential evaluation.
- Figures: Figure 1 reports best-of-three union results for each model-bug pair with vendor-grouped columns.The visualization compares how many bugs reach successive capability tiers.
- Figures: Figure 2 tracks cumulative capability unlock over turns for each bug-seed pair on 12 bugs with the most Tier-3-or-higher activity.The traces compare progression and saturation timing across agents.
- Figures: Figure 3 measures the conditional probability of advancing one capability step in the primary arm.The figure groups agents by where their progression stalls in the ladder.
- Figures: Figure 4 reports mean capability count over turns by V8 subsystem with shaded ±1 SEM.The subsystem comparison separates WebAssembly, JavaScript, and JIT-compiler bugs.
- Measurement arms: Figure 5 compares bugs reaching each tier across the primary, coaching, and vendor-CLI measurement arms.The vendor-CLI arm is run only for GPT-5.5, while the other arms assess harness sensitivity across models.
- Cost analysis: Figure 6 plots per-episode cost against mean capability score across three seeds, using color for model and shape for arm.The x-axis is logarithmic, and the dashed line marks the Pareto frontier.
- Trajectory analysis: Figure 7 presents a primitive-by-primitive trajectory for a Mythos Preview run that reaches ACE on a WebAssembly type-confusion bug.The plot separates flags reached by other models from those reached only by Mythos Preview in the primary arm.