Source-linked AI summary
EVMbench: Evaluating AI Agents on Smart Contract Security
Justin Wang, Andreas Bigger, Xiaohai Xu, Justin W. Lin, Andy Applebaum, Tejal Patwardhan, Alpin Yukseloglu, Olivia Watkins
TL;DR
Smart-contract vulnerabilities threaten substantial on-chain value, raising the question of how well AI agents can discover, repair, and exploit them. EVMbench answers this with a curated, programmatically graded benchmark spanning three task modes in local Ethereum environments. Frontier agents achieve end-to-end exploitation and strong task-specific performance, while results show that coverage, information access, and scaffolding materially shape outcomes.
Problem
Existing evaluations do not measure the full cycle of discovering, mitigating, and exploiting real smart-contract vulnerabilities, despite the high stakes of on-chain assets.
Method
EVMbench evaluates detection, patching, and exploitation using curated high-severity vulnerabilities, programmatic grading, and local Ethereum execution.
Results
Agents can execute end-to-end exploits, but performance emphasizes depth of review and is materially affected by hints and scaffolding.
Takeaways & Limitations
The released benchmark supports reproducible measurement of exploit capability, failure modes, and defensive progress in smart-contract security.
Takeaways & Limitations
The curated vulnerabilities are not representative of the full space of real-world blockchain bugs, and grading cannot credit undisclosed findings or predefined-exploit alternatives.
Abstract
from arXiv · showhide
Smart contracts on public blockchains now manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can already navigate this landscape, both in ways that improve security and in ways that might increase risk. We introduce EVMbench, an evaluation that measures the ability of agents to detect, patch, and exploit smart contract vulnerabilities. EVMbench draws on 117 curated vulnerabilities from 40 repositories and, in the most realistic setting, uses programmatic grading based on tests and blockchain state under a local Ethereum execution environment. We evaluate a range of frontier agents and find that they are capable of discovering and exploiting vulnerabilities end-to-end against live blockchain instances. We release code, tasks, and tooling to support continued measurement of these capabilities and future work on security.
1 Introduction
EVMbench addresses the need to measure AI agents’ smart-contract security capabilities across discovery, mitigation, and exploitation. It evaluates these capabilities in realistic, reproducible environments and releases the benchmark for continued measurement.
- Motivation: Public blockchains secure substantial assets, while smart-contract exploits can cause immediate and immense consequences.Stablecoins settle trillions of dollars monthly, and more than $100B of assets routinely sit in open-source smart contracts.
- Motivation: Existing evaluations do not cover the full cycle of discovering, mitigating, and exploiting real smart-contract vulnerabilities.Prior work largely targets traditional cybersecurity, narrow subtasks, or simplified environments.
- Benchmark: EVMbench evaluates agents on detecting, patching, and exploiting fund-draining vulnerabilities in production-grade smart-contract environments.The benchmark emphasizes comprehensive coverage rather than stopping after a single successful finding.
- Benchmark: Detect measures recall of ground-truth vulnerabilities, Patch checks preserved tests and failed exploits, and Exploit grades transactions against contracts and balances.Exploit transactions are replayed in an isolated grader container against an Ethereum instance.
- Contributions: The benchmark curates high-severity vulnerabilities from real audits, supplies oracle patches and exploits, and preserves realistic codebases, constraints, and attacker incentives.The authors evaluate frontier agents across all three modes and analyze their rollouts and failure modes.
- Contributions: The authors release EVMbench’s dataset and evaluation harness to support further capability measurement and security research.The release is intended to encourage future work on security.
2 Background
Smart contracts run on deterministic, stateful blockchain infrastructure that supports economically meaningful applications and machine-checkable evaluation. These same properties make security failures costly and motivate end-to-end AI-agent evaluation.
- Smart Contracts and the EVM: Smart contracts underpin applications such as automated exchanges and lending markets, securing billions of dollars in assets.They are commonly written in Solidity, compiled to bytecode, and deployed at fixed addresses.
- Smart Contracts and the EVM: The EVM processes transactions deterministically, maintaining account balances, contract code, persistent storage, call frames, and emitted logs.Given the same starting state and transaction sequence, every node computes the same result.
- Security and Audits: Blockchain immutability and permissionless execution can turn smart-contract vulnerabilities into instant, irreversible fund loss.Competitive audits attempt to surface impactful vulnerabilities before deployment.
- Security and Audits: Ethereum’s adversarial environment requires reasoning about economic incentives, stateful multi-step interactions, protocol composition, and other actors’ behavior.Pending transactions can be publicly visible, enabling automated searchers and bots to scan for and front-run opportunities.
- Evaluation Environment: Deterministic replay, machine-checkable success signals, and sandboxed isolation make blockchains suitable for evaluating end-to-end agent behavior.Success signals include account-balance changes and emitted events, while effects remain confined to an isolated environment.
3 Evaluation
EVMbench builds a curated, multi-mode evaluation around high-severity audit vulnerabilities and grades agents through reports, code tests, and blockchain-state checks. Its environments prioritize realistic tasks, reproducibility, and resistance to cheating.
- 3.1 Task Curation: The benchmark contains 117 vulnerabilities from 40 audits, filtered toward high-severity issues that can directly cause user or platform fund loss.Additional scenarios come from the Tempo blockchain’s agentic stablecoin-payment auditing process.
- 3 Evaluation: EVMbench supports Detect, Patch, and Exploit modes with distinct grading methods and a shared requirement for comprehensive vulnerability coverage.The most realistic mode gives agents a funded wallet and evaluates end-to-end exploits.
- Evaluation Framework: The evaluation runs in isolated containers and uses hardened, reproducible infrastructure with manual quality control of tasks, rollouts, and oracle grading.A Rust-based program manages blockchain setup, deployment, and grading; a JSON-RPC gatekeeper helps prevent invalid-method cheating.
- 3.2.1 Detect: Detect scores the percentage of ground-truth vulnerabilities identified in an audit report using a model-based judge.Agents also receive a financial award based on historical audit payouts.
- 3.2.2 Patch: Patch agents modify vulnerable repositories directly, while grading checks that applicable existing tests still pass after changes.Tests depending on vulnerable logic are exempted, and test files are reset before execution to prevent cheating.
- 3.2.3 Exploit: Exploit agents analyze a deployed local Ethereum instance, optionally deploy helper contracts, and craft transactions to realize attacks end-to-end.Custom scripts re-deploy contracts, replay transactions, and inspect blockchain state for vulnerability-specific success.
4 Experiments and Results
EVMbench evaluates agents across detection, patching, and end-to-end exploitation, including realistic flash-loan attacks. Results show strong but uneven performance, with hints improving repair and exploitation while vulnerability discovery remains difficult.
- Evaluation setup: Agents were evaluated on Detect, Patch, and Exploit tasks using curated vulnerabilities and increasingly realistic exploit setups.Patch and Exploit evaluations included manually configured vulnerabilities, while the exploit scenario connected agents to local blockchain state.
- Quantitative results: GPT-5.3-Codex scored highest on Patch at 41.7% and Exploit at 71.0%, while Claude Opus 4.6 led Detect at 45.9%.GPT-5.2 performed better through Codex CLI than through OpenCode.
- Quantitative results: Claude Opus 4.6 achieved the highest vulnerability payout, but all models remained far below the attainable audit award.A small number of vulnerabilities account for a large share of the total reward.
- Quantitative results: GPT-5.3-Codex and GPT-5.2 scored highly while reporting vulnerability counts close to the dataset ground truth.The benchmark notes that real audit codebases may contain undiscovered vulnerabilities outside the ground truth.
- Hint experiments: With mechanism hints, GPT-5.2 reached 90.2% on Patch and 78.3% on Exploit, indicating discovery is a major bottleneck in large repositories.Agents generally knew how to repair exposed Solidity mechanisms but sometimes struggled to construct the transactions needed for exploitation.
- Exploit rollout: In the NOYA H-08 example, GPT-5.2 identified missing access control, encoded a two-call payload, executed the flash loan, and verified the victim balance was drained.The attack used the registry and flash-loan pathway to move tokens from the victim connector to the attacker wallet.
5 Discussion
The discussion frames EVMbench as a measure of review coverage and end-to-end exploit capability, not merely isolated bug-finding. It also shows that hints, scaffolds, and workflow choices materially affect measured performance.
- Coverage: EVMbench emphasizes depth and coverage of review because agents can find or exploit individual bugs while missing critical vulnerabilities in the same codebase.The benchmark therefore measures more than isolated successes.
- Bottlenecks: Hints sharply improve Patch and Exploit performance, suggesting vulnerability discovery is often the primary bottleneck rather than repair or transaction construction.The experiments separate model capability from information access and scaffolding.
- Scaffolding: Stronger scaffolds can outperform OpenCode even with similar model families, showing that tooling and workflow choices materially change effective agent performance.The discussion recommends reporting both model capability and agent design choices.
- Security implications: The best agents can execute end-to-end exploits against a nontrivial fraction of vulnerable contracts, making reproducible exploit measurement important for tracking progress and failure modes.Such exploits can translate directly into transferable value.
6 Limitations
EVMbench’s measurements are constrained by curated, predefined vulnerability targets and by simplified execution environments. These choices limit coverage of real-world bugs, timing behavior, chain state, and cross-chain interactions.
- The evaluated vulnerabilities are curated from Code4rena findings and are not representative of the full space of real-world bugs.Results should not be read as the fraction of all live blockchain bugs that are exploitable.
- Detect can only credit vulnerabilities already identified by auditors, while Patch and Exploit grade only predefined exploits.Undiscovered valid issues cannot receive credit in Detect, and additional vulnerabilities may distract agents in Patch and Exploit.
- EVMbench contains only 44 patch and 23 exploit vulnerabilities because validating tasks across codebases and protocols is time-intensive.
- Exploit Mode Limitations: Exploit tasks exclude correctness conditions requiring complex timing because replay timestamps are sequential and do not exactly reproduce the originals.
- Exploit Mode Limitations: Exploit tasks begin from empty local chain storage and cannot reproduce forks, precompiles, or historical transactions.Protocols therefore execute atop a blank local chain state.
- Exploit Mode Limitations: The single-chain environment excludes cross-chain interactions and requires mock contracts when production counterparts exist only on Ethereum Mainnet.Agents may spend reasoning effort examining mock ownership or permissions that would be less relevant on a forked chain.
7 Related Work
Prior AI security evaluations use CTFs, historical CVEs, and real-world vulnerability reproduction, but often cover only parts of the vulnerability lifecycle. Smart-contract benchmarks connect security evaluation with economic impact while differing in environment and task coverage.
- Existing AI cybersecurity evaluations include professional CTF suites, historical-CVE benchmarks, and vulnerability reproduction over real-world projects.These benchmarks commonly generate proof-of-concept tests from vulnerability descriptions and pre-patch codebases.
- Crypto provides a direct economic-impact setting because smart-contract exploits can translate immediately into real financial losses.Related evaluations also incorporate economic metrics for software engineering, security vulnerabilities, and economically valuable tasks.
- Smart Contracts: SCONE-Bench evaluates exploit-only tasks on a forked blockchain, whereas EVMbench uses a fresh blockchain and additionally evaluates Detect and Patch.The fresh-chain design improves control and reproducibility but omits real mainnet history.
8 Conclusion
EVMbench introduces a framework for measuring agents’ ability to detect, patch, and exploit smart-contract vulnerabilities in realistic environments. The authors position it as a basis for tracking these capabilities and supporting responsible security work.
- EVMbench measures agents’ ability to detect, patch, and exploit smart-contract vulnerabilities using high-severity issues from competitive audits.The benchmark evaluates end-to-end behavior in realistic environments.
- The authors aim for EVMbench to support progress tracking, capability understanding, and responsible adoption of AI for vulnerability discovery and patching.They describe this support as important for improving security at scale.
9 Code and Data
The paper releases EVMbench code and data while identifying important extensions beyond its current Solidity and EVM focus. It points to execution clients, transaction landing, cryptographic systems, cross-chain protocols, stablecoins, and DeFi as further evaluation targets.
- Code and Data: Code and data for running EVMbench are available at the project’s GitHub repository.
- Code and Data: The release includes canary strings in scripts and hint files to help researchers filter evaluation samples from future training data.Publications reproducing excerpts should include the canary string for the same filtering purpose.
- Future Directions: EVMbench focuses on Solidity smart contracts running on EVM-compatible chains and does not cover the full crypto security surface.Solana and specialized EVM chains are identified as promising future directions because their programming models or features may create distinct failure modes.
- Future Directions: Evaluating execution-client bugs is a high-stakes extension because clients secure hundreds of billions of dollars, although their codebases are massive and bugs rarer.
- Future Directions: Transaction-landing evaluations would cover MEV behaviors such as sandwich attacks, frontrunning, and backrunning that depend on mempools, builders, and timing.Reliable transaction landing is also described as relevant to economically valuable production crypto work.
- Future Directions: Zero-knowledge circuits, threshold signatures, MPC protocols, and FHE schemes are additional security-critical cryptographic systems for evaluation.The paper notes that zero-knowledge bugs are difficult to detect through testing because these systems are designed for obfuscation and privacy.
- Future Directions: The current single-chain setup does not capture cross-chain bridge exploits, despite their history among major crypto hacks.The attack surface is growing as more chains launch and interoperability becomes standard.
- Future Directions: Stablecoin payments and DeFi transactions are proposed as economically meaningful, measurable evaluation areas beyond security.These tasks involve concrete success criteria such as completed payments, executed trades, route quality, and healthy loans.
B Task Curation and Dataset Information
EVMbench is constructed from curated high-severity, fund-loss vulnerabilities in real audit repositories, with oracle patches and exploits validating Patch and Exploit tasks. The benchmark combines 117 Detect vulnerabilities with manually configured subsets for Patch and Exploit, while preserving realistic codebases and programmatic blockchain-based grading.
- Dataset construction: High-severity Code4rena findings were filtered for potential fund loss, Solidity implementation, post-July-2023 discovery, and redistributable licensing.The curation process included automated classification followed by manual filtering and quality control.
- Validation: Oracle patches were accepted only when unaffected original tests passed and the corresponding exploit reproduction failed.This validation checks both preservation of expected behavior and removal of the exploit path.
- Validation: Oracle exploits encode attacker behavior in commands or helper contracts, execute transaction sequences, and require a grader success signal after exploitation.Exploit validation therefore covers the full on-chain sequence rather than only a code-level trigger.
- Detect awards: Detect awards use the median historical payout for each vulnerability, making the score a lower-bound reward estimate.Vulnerabilities without payout information receive $0, and the 10 highest-paying vulnerabilities account for 72.94% of total award mass.
- Task characteristics: Many Patch vulnerabilities require changes of 5 or fewer lines, while only 4 require more than 20 changed lines; oracle exploits require no large transaction count.These distributions characterize the benchmark’s patch and exploit complexity.
H.2 Additional Qualitative Analysis
Qualitative examples show that agents can identify plausible vulnerabilities and sometimes produce successful patches or detection reports, yet end-to-end exploitation remains vulnerable to compilation, balance-checking, and execution errors. Common failures also include targeting the wrong issue, leaving vulnerabilities partially fixed, preserving an exploit path, or breaking tests.
- Exploit analysis: GPT-5 identifies a plausible Phi Protocol reentrancy surface but fails to compile and execute a correct end-to-end exploit.It also fails to notice that exploit transactions cost funds from its own wallet.
- Patch analysis: GPT-5 successfully patches the Phi H-06 reentrancy by adding nonReentrant; existing tests pass and the exploit test fails.This satisfies the benchmark’s patch-award conditions in the example.
- Detect and exploit analysis: GPT-5.1 identifies both Basin ground-truth vulnerabilities in Detect, but its corresponding Exploit rollout fails after it drains tokens the target never held.The agent does not check the contract’s ETH balance before concluding that the exploit succeeded.
- Common failure modes: Detect can score zero even when an agent finds a legitimate issue because grading requires matching the specific ground-truth vulnerability ID.Broad thematic audits may omit the concrete, scoped vulnerability used for scoring.
- Common failure modes: Patch failures include addressing the wrong issue, fixing only part of a multi-vulnerability repository, leaving the exploit path viable, or breaking existing tests.These failure modes distinguish local code changes from complete, graded remediation.
- Common failure modes: Exploit failures include invalid end-to-end action sequences, submitting no transactions, or abandoning the attempt before completion.Plausible reasoning alone does not earn credit without a valid executed exploit.
J Full Results
The full-results section reports bootstrap score estimates with 95% confidence intervals across evaluated models and task modes. It also breaks out GPT-5.2 results by hint level and identifies the model snapshots and runtime versions used in the experiments.
- Full results: 95% bootstrap confidence intervals accompany score estimates across the evaluated models and task modes.The intervals use N = 10,000 resamples and separate estimate and confidence-interval columns.
- Full results: Table 9 reports full results across all task modes.The section’s model and runtime details are documented separately through model aliases, snapshots, and CLI versions.