EVMbench: Evaluating AI Agents on Smart Contract Security

1 Introduction

• The benchmark addresses a gap in prior work, which focused on traditional cyber vulnerabilities or narrow subtasks and simplified environments rather than the full cycle of real-vulnerability discovery, mitigation, and exploitation.The authors argue that deterministic execution, programmatic verification, and isolated environments make blockchains a clean yet realistic evaluation surface.Source: §1 · p. 1
• EVMbench evaluates detection, patching, and end-to-end exploitation of fund-draining vulnerabilities, with grading based on on-chain events and balance changes in Exploit mode.The benchmark also measures comprehensive coverage: agents must address all relevant vulnerabilities in a codebase, not just achieve one successful fix or exploit.Source: §1 · pp. 1–2, Figure 1 · p. 2
• A Rust re-execution framework replays agent transactions against a local Ethereum node for fast, reproducible Exploit evaluation while guarding against cheating.Exploit grading replays transactions in an isolated grader and checks vulnerability-specific contract and balance outcomes.Source: §1 · p. 2, Figure 1 · p. 2
• The task suite uses high-severity vulnerabilities from real audits, oracle patches and exploits, and realistic codebases, constraints, and attacker incentives.The dataset and evaluation harness are open-sourced to support further measurement and security research.Source: §1 · p. 2
• The authors evaluate frontier agents across all three modes, compare performance, and analyze rollouts and failure modes.The motivation is that smart contracts hold substantial assets, while their speed and irreversibility can make exploits immediate and consequential.Source: §1 · pp. 1–2

2.1 Smart Contracts and the EVM

• Solidity contracts compile to bytecode at fixed addresses, accept function-call transactions, and underpin exchanges and lending markets securing billions.The EVM executes transactions deterministically and maintains global account balances, contract code, and persistent storage.Source: §2.1 · p. 3

2.2 Smart Contract Exploits and Audits in Practice

• Smart contract vulnerabilities can cause instant, irreversible fund loss, prompting time-limited competitive audits, yet profitable attacks can be rapidly front-run or copied.Pending Ethereum transactions are publicly visible, and automated searchers and bots continuously scan for opportunities.Source: §2.2 · pp. 3–4

2.3 Blockchains as an Evaluation Substrate

• Deterministic replay, verifiable success signals, and sandboxed real-protocol deployments make blockchains a natural substrate for end-to-end agent benchmarks.The “dark forest” setting also challenges agents to reason about incentives, stateful interactions, protocol composition, and other actors; EVMbench uses these properties to evaluate detection, patching, and exploitation.Source: §2.3 · p. 4

3.1 Task Curation

• The task set contains 117 vulnerabilities from 40 audits, filters for high-severity fund-loss risks, and includes Tempo stablecoin-payment scenarios.Most vulnerabilities come from Code4rena audits; the Tempo scenarios ground the benchmark in agentic stablecoin payments.Source: §3.1 · pp. 4–5

3.2 Harness and Evaluation Modes

• Three audit-specific modes assess security skills, requiring agents to address all high-severity vulnerabilities; the most realistic provides a funded wallet for end-to-end exploits on a local blockchain.Agents work in isolated Ubuntu 24.04 Docker containers without web or internet access, while grading runs separately.Source: §3.2 · p. 5

3.2.1 Detect

• Detect mode scores the percentage of ground-truth vulnerabilities identified in an audit report, using a model-based judge, and assigns awards based on historical audit payouts.The objective is to identify high-severity vulnerabilities that could cause loss of user or platform funds.Source: §3.2.1 · p. 5

3.2.2 Patch

• Patch mode checks that existing tests still pass where appropriate and uses unseen exploit tests to verify each vulnerability is no longer exploitable.Before testing, graders reset test files the agent was not allowed to modify; failures in tests relying on vulnerable logic are not penalized.Source: §3.2.2 · p. 5

3.2.3 Exploit

• Exploit mode gives agents a funded key and RPC endpoint to analyze the chain, deploy helpers if needed, and send transactions; custom scripts grade replayed transactions against blockchain state.A Rust-based evaluation program manages the local chain, contract deployment, and grading via chain analysis.Source: §3.2.3 · pp. 5–6

3.3 Ensuring Benchmark Reliability

• Reliability measures include filtering severe, reproducible tasks, removing environment-specific failures, multiple rounds of quality control, and hardening exploit execution behind a JSON-RPC gatekeeper.Quality checks include manual review for cheating or patching hacks and validation that graders score oracle solutions correctly.Source: §3.3 · p. 6

4 Experiments and Results

• Detect covered all 117 vulnerabilities, while Patch and Exploit used manually configured sets of 44 and 23 vulnerabilities from 22 and 16 repositories, respectively.Patch and Exploit often required extensive manual setup and validation; OpenCode’s basic ReAct loop served as a baseline against more advanced scaffolds such as Codex CLI.Source: §4 · p. 6

4.1 Quantitative Results

• GPT-5.3-Codex led Patch (41.7%) and Exploit (71.0%), while Claude Opus 4.6 led Detect (45.9%); GPT-5.2 scored higher via Codex CLI than OpenCode.GPT-5.3-Codex achieved its high performance while remaining token-efficient.Source: §4.1 · p. 7, Figure 4 · p. 7
• Claude Opus 4.6 achieved the highest Detect vulnerability payout, but all models fell far short of the attainable award.A small number of vulnerabilities account for a large share of the reward, so finding one can dramatically increase an agent’s payout.Source: §4.1 · p. 7, Table 2 · p. 7
• GPT-5.3-Codex and GPT-5.2 scored highly while reporting vulnerability counts similar to ground truth, which the authors relate to competitive audits’ coverage of high-severity issues.The comparison concerns disclosure volume and mean vulnerability disclosure count.Source: §4.1 · p. 7, Figure 5 · p. 8
• With medium mechanism hints, GPT-5.2 scored 90.2% on Patch and 78.3% on Exploit; Patch difficulty largely reflected discovery, while agents still struggled to craft exploit transactions.The hint experiments also tested low-level file hints and high-level grader-check hints.Source: §4 · pp. 6–7, §4.1 · p. 7, Figure 7 · p. 10

4.2.1 GPT-5.2 Exploits a Fund Drain End-To-End Using a Flash Loan

• GPT-5.2 exploited NOYA H-08 by abusing unauthenticated registry flash-loan routing to relay victim-token transfers through a two-call payload, then verified the drain.The task graded the post-exploit balance of the agent’s wallet; draining the victim allocation earned a full score. The rollout is an example of the successful exploits discussed in the section.Source: §4.2 · p. 7, §4.2.1 · p. 8, Figure 6 · p. 9

• High scores on specific audits can coexist with missed critical vulnerabilities in the same codebase, so EVMbench measures review depth under constraints rather than isolated wins.This gap persists even when a model can exploit a vulnerability end-to-end.Source: §5 · p. 10
• Hints sharply improve Patch and Exploit performance, suggesting discovery is often the primary bottleneck, while tooling and workflow choices materially affect outcomes.The OpenCode baseline trails stronger scaffolds even with similar model families; evaluations should report both model capability and agent design choices.Source: §5 · p. 10
• Despite imperfect coverage, the best agents can execute end-to-end exploits against a nontrivial fraction of vulnerable contracts, making precise, reproducible measurement urgent.Because exploits can translate directly into transferable value, improvements in models and scaffolds raise the expected attack surface; measurement can track progress and guide mitigations.Source: §5 · p. 10

6 Limitations

• The curated Code4rena vulnerabilities are not representative of all real-world bugs, so results cannot be read as the share of live blockchain bugs that are exploitable.Source: §6 · p. 10
• Detect mode cannot credit valid issues absent from historical reports, while unvalidated submissions can be false positives; Patch and Exploit grade only predefined exploits.Source: §6 · p. 10
• EVMbench includes only 44 patch and 23 exploit vulnerabilities because setting up and validating tasks across codebases and protocols is time-intensive.Source: §6 · p. 10

6.1 Exploit Mode Limitations

• Exploit tasks exclude correctness dependent on complex timing because sequential replay does not reproduce original transaction timestamps exactly.Source: §6.1 · p. 11
• Exploit tasks run on fresh Anvil chains with configurable parameters but empty storage, without forking, pre-compiles, or historical transactions.Protocols therefore execute atop a blank local chain state.Source: §6.1 · p. 11
• Exploit supports only one chain, requiring mocks for some production contracts and leading agents to spend reasoning cycles examining mock ownership or permissions.Cross-chain interactions are out of scope; the authors observed this mock-focused behavior would be less likely on a forked chain with real deployed contracts.Source: §6.1 · p. 11

Security

• EVMbench evaluates Detect, Exploit, and Patch on smart contracts, extending lifecycle-based security evaluation with on-chain state transitions, funded attacker wallets, adversarial transactions, and EVM-native execution.Prior evaluations include CTF suites, historical-CVE benchmarks, CyberGym, and BountyBench’s lifecycle evaluation on systems with validated bug bounties.Source: §7 · p. 11

Economic Impact

• Crypto offers a direct setting for studying economic impact because malicious smart-contract exploits can translate immediately into financial losses; prior work measures economic impacts across software engineering, security vulnerabilities, and other valuable tasks.This interest arises amid rapid proliferation of AI tools and capability improvements.Source: §7 · p. 11

Smart Contracts

• Unlike SCONE-Bench’s exploit-only tasks on a forked blockchain, EVMbench tests Detect and Patch as well as Exploit on fresh chains without historical state, improving control and reproducibility while omitting mainnet history.EVMbench grades with task-specific scripts, including balance checks when appropriate; SCONE-Bench grades via balance checks.Source: §7 · p. 11

8 Conclusion

• EVMbench instantiates a framework for measuring AI agents’ ability to detect, patch, and exploit smart contract vulnerabilities.The authors present it as a basis for tracking progress and understanding these capabilities.Source: §8 · p. 11
• The benchmark uses high-severity vulnerabilities found in competitive audits and evaluates end-to-end behavior in realistic environments.The authors describe these capabilities as difficult and important to measure given the scale of on-chain assets.Source: §8 · p. 11
• The authors hope releasing EVMbench supports responsible AI adoption for vulnerability discovery and patching in practice, which they view as critical to improving security at scale.Source: §8 · p. 12

9 Code and Data

• EVMbench code and data are available on GitHub, and the paper recommends including its canary string in published excerpts to help filter evaluation samples from future training data.The canary is also included in scripts and hint files.Source: §9 · p. 12

A Future Directions

• EVMbench’s Solidity and EVM-compatible-chain focus reflects current auditing activity but does not cover the full crypto security surface.Source: A Future Directions · p. 14

A.1 Expanding Chain Coverage

• The paper identifies Solana and specialized EVM chains such as Tempo as expansion targets because their distinct programming models or custom features may introduce different vulnerability classes.Solana is described as the second-largest smart contract platform by TVL; its Rust, accounts, and program-derived addresses differ substantially from the EVM.Source: §A.1 · p. 14

A.2 Protocol and Node Security

• Evaluating execution-client security in geth, reth, and equivalents could address catastrophic state-transition bugs, despite massive codebases, rare bugs, and demanding reproduction infrastructure.These clients collectively secure hundreds of billions of dollars in market cap.Source: §A.2 · p. 14

A.3 Mempool and Transaction Ordering

• Many exploits depend on transaction ordering, so evaluating mempool, block-builder, and timing reasoning could assess agents’ ability to land transactions reliably in production crypto environments.The passage names sandwich attacks, frontrunning, and backrunning as examples of MEV-related activity.Source: §A.3 · p. 14

A.4 Private Chains and Cryptographic Vulnerabilities

• Zero-knowledge circuit bugs are difficult to detect through testing, and the paper also identifies threshold signatures, MPC, and FHE as important production cryptographic systems.The passage emphasizes the stakes of finding vulnerabilities during development because these systems rely on obfuscation and privacy.Source: §A.4 · p. 14

A.5 Cross-Chain Protocols and Bridges

• Bridge exploits such as Ronin, Wormhole, and Nomad illustrate a cross-chain attack surface that EVMbench’s single-chain setup does not capture.Cross-chain protocols combine multiple execution environments with message-passing and trust assumptions, and the attack surface grows as interoperability becomes standard.Source: §A.5 · p. 14

A.6 Beyond Security: Crypto as an Economic Evaluation Substrate

• Crypto evaluations could extend beyond security to reliable stablecoin payments and multi-step DeFi transactions with measurable execution, price, and loan-health outcomes.The paper notes that security is only a small portion of economically valuable crypto work. Payment tasks involve transaction construction, mempool propagation, gas, nonces, and contract interactions; DeFi tasks include DEX routing, borrowing, and liquidation-risk management.Source: §A.6 · p. 14

B.1.1 Selection / B.1.2 Creation

• The final set contains 117 filtered vulnerabilities; Patch covers 44, and Exploit covers 23 selected from the Patch set.Selection retained high-severity fund-loss Solidity vulnerabilities discovered after July 2023 from repositories with redistribution permissions, followed by manual quality control. Tasks often used auditor-provided tests, mitigations, or deployment scripts, which the authors supplied or repaired when needed.Source: §B.1.1 · p. 15, §B.1.2 · p. 15

B.1.3 Initial Validation

• Oracle patches must preserve unrelated tests and make the exploit reproduction fail; oracle exploits execute transaction sequences that the grader recognizes as successful.Exploit validation included encoding the exploit pattern and checking that the grader fails when exploitation does not succeed.Source: §B.1.3 · p. 15

B.2 Additional Dataset Information / B.2.1 Detect

• Among Code4rena audits with nonzero awards, award size was weakly negatively associated with the number of distinct auditors who found an issue (Pearson r = −0.28, n = 92).Detect awards use median auditor rewards per vulnerability, a lower bound because early discoverers may receive bonuses. Invitational contests without payout information and Tempo tasks are assigned $0; the ten highest-paying vulnerabilities account for 72.94% of total award mass.Source: §B.2.1 · pp. 15, 16, 18, Table 3 · p. 16

B.2.2 Patch / B.2.3 Exploit / B.2.4 Example Oracle Solutions

• Many oracle patches change five or fewer lines, only four change more than 20, and oracle exploits require no large number of transactions.Examples show an access-control patch adding onlyOwner to authorizeUpgrade and an exploit using a single isolateLiquidate call to liquidate collateral the attacker does not own.Source: §B.2.2 · p. 18, Figure 10 · p. 18, §B.2.3 · p. 18

C Detect Grading

• The Detect judge, GPT-5, assesses whether an agent report meaningfully identifies each ground-truth vulnerability, with stress tests for under-crediting, over-crediting, and prompt injection.The tests check that slight technical errors do not prevent credit, while meaningfully incorrect findings and reports that claim to contain ground truth are not credited.Source: C Detect Grading · p. 20

D Blockchain Tooling / D.1 Execution Environment / D.2 Ploit / D.3 Veto / E Environment Hardening

• Exploit tasks run on deterministic, pre-funded Anvil chains; ploit orchestrates local-chain tasks, while veto blocks simulator-specific RPC methods to preserve a standard Ethereum attack surface.Environment hardening replaces Anvil’s known default mnemonic with a custom one and removes deployment scripts that could reveal vulnerabilities.Source: §D.1 · p. 22, §D.2 · p. 22, §D.3 · p. 22

F Hints / G Prompting

• Hints range from a location only, to location plus mechanism, to those hints plus grading information; Detect and Patch use low and medium levels, while Exploit uses all three.Hints aim to narrow search without providing a full step-by-step solution, and the paper provides the prompts used for each task mode.Source: F Hints · p. 23, G Prompting · p. 23

H Qualitative Analysis

• The qualitative analysis presents agent submissions and rollout analysis for 2024-08-phi H-06, plus commentary on other rollouts, including GPT-5.1 examples used in pre-final manual task quality control.Source: H Qualitative Analysis · p. 25

H.1 2024-08-phi H-06 Analysis

• H-06 exploits ETH refunds before state updates: reentry reuses a credId to buy on a cheap curve, switch metadata to an expensive curve, and sell, siphoning curator liquidity.The recommended fix is to make createCred and its callees nonReentrant and apply checks-effects-interactions so refunds follow critical state transitions.Source: §H.1 · p. 25
• The exploit task seeds 50 ETH, leaks valid signatures in reverted transaction calldata, and scores the fraction of Cred ETH drained down to a 0.1 ETH threshold.Solvers must recover the signed data, configure a helper proxy, and reenter to trade across whitelisted cheap and expensive curves.Source: §H.1 · p. 25
• GPT-5 identified a plausible Phi reentrancy surface but failed to execute the exploit end to end and did not notice its attempted transactions lost funds.The rollout shows the agent did not check its wallet ETH balance before sending exploit transactions.Source: §H.1 · p. 25, Figure 19 · p. 26

H.2 Additional Qualitative Analysis

• GPT-5 detected Phi H-06 and patched reentrancy with nonReentrant; existing tests passed and the exploit test failed after the change.Its Detect submission listed H-06 as C-03.Source: Figure 20 · p. 27, Figure 21 · p. 28
• On 2024-07-basin, GPT-5.1 identified both ground-truth vulnerabilities in Detect but failed to exploit the WellUpgradeable error, misreading token balances as success.The rollout shows it deployed a malicious contract and called the drain method but did not check the contract’s ETH balance.Source: §H.2 · p. 25, Figure 22 · p. 29, Figure 23 · p. 30

H.3 Failure Modes

• Detect failures arise when reports discuss broad themes or other valid loss-of-funds issues but omit the specific ground-truth vulnerability ID, producing zero or partial coverage.Exploit tasks instead reward direct operational success.Source: §H.3 · p. 31

Patch.

• Patch failures include targeting the wrong bug, leaving graded bugs unfixed, failing to block the exploit path, or breaking existing tests.Source: Patch · p. 31

Exploit.

• Exploit failures include an incomplete or incorrect exploit path, submitting no transactions, or giving up before completion.Source: Exploit · p. 31

J Full Results

• Confidence intervals are computed from 10,000 bootstrap resamples and reported at the 95% level.Source: J Full Results · p. 32
• GPT-5.2 (xhigh reasoning) results are reported by hint level.Source: Table 8 · p. 32
• Full results cover all task modes and report bootstrap score estimates with separate 95% confidence intervals.Table 9 gives the full-results scope; confidence intervals use 10,000 bootstrap resamples.Source: Table 9 · p. 32, J Full Results · p. 32