Source-linked AI summary
CWE-bench: Refocusing Agentic Security from Exploitation to Defense
Gonzalo Gonzalez-Pumariega, Ajay Jagannath, Adit Jain, Guram Gogia, Soumyadeep Bakshi, Nazneen Rajani
TL;DR
Defensive agent evaluations have received less attention than offensive ones, despite the challenge of finding and fixing every vulnerability without breaking software. CWE-bench tests agents on private, unlabeled repository repairs and validates its grading. The best agents reach 68% pass@1 and 81% pass@4, while performance varies across retries, costs, languages, and vulnerability categories.
Problem
Cybersecurity evaluations have focused largely on offense, while evidence on whether agents can find and repair unknown vulnerabilities without breaking software is more limited.
Source: §1 · p. 1Method
CWE-bench evaluates agents on 120 private repository audit-and-repair tasks with no diagnosis provided, requiring patches to block exploits and preserve functionality.
Source: §1 · pp. 1–2Results
68% pass@1 and 81% pass@4 are the best agents’ scores; agents differ in retry gains and cost, and difficulty varies more by vulnerability category than by language.
Source: §1 · p. 2Takeaways & Limitations
A single pass rate hides differences in retry response, task coverage, cost, and vulnerability classes handled, so defensive capability must be measured across vulnerability breadth.
Source: §6 · p. 7CWE-bench covers 73 CWEs, a small fraction of the more than 900 catalogued weaknesses, so broader class coverage and per-class reporting are needed to judge readiness.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a private benchmark of 120 held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all OWASP Top 10 2025 categories. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a programmatic verifier. We validated the verifier against a panel of agentic judges, manually reviewing disagreements during development and refining the checks. The best agents reach 68% pass@1 and 81% pass@4. We further conduct analyses showing that (i) agents with the same pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or vulnerability categories, and difficulty varies far more by vulnerability category than by language. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.
1 Introduction
CWE-bench addresses the gap between offensive evaluations and the harder task of finding and fixing all vulnerabilities without breaking software. It evaluates agents on private, unlabeled repository repairs and reports substantial differences beyond pass@1.
1 Introduction
- Unlike offensive evaluations, which can be validated by a successful exploit, defense requires finding and fixing every vulnerability while preserving the software’s intended function.The introduction illustrates this with path traversal: disabling downloads would remove the service the file server is meant to provide.Source: §1 · p. 1
- CWE-bench comprises 120 private held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all ten OWASP Top 10:2025 categories.Agents receive a repository and threat-model instruction, without being told the weakness, its location, or the number of vulnerable sites.Source: §1 · pp. 1–2
- Prior defensive benchmarks vary in task provenance, diagnosis supplied to agents, and weakness coverage, motivating a benchmark spanning real repositories and broad vulnerability categories.The comparison covers recent vulnerability-repair benchmarks, with some table values derived by the authors.Source: §1 · p. 1, Table 1 · p. 2
- The verifiers were compared with three agentic judges, and every disagreement during development was manually reviewed to refine the checks.A patch must block exploits while preserving original functionality.Source: §1 · p. 2
- 68% pass@1 and 81% pass@4 are the best agents’ results; agents with equal pass@1 differ in retry gains and cost, and no single agent leads across languages or vulnerability categories.Difficulty varies more across vulnerability categories than across languages.Source: §1 · p. 2
2 Related Work
Prior cybersecurity evaluations emphasize offense, while defensive benchmarks often simplify auditing or rely on checks that can misgrade patches. CWE-bench combines unlabeled repair tasks with programmatic grading and judge review.
2 Related Work
- Offensive benchmarks show whether agents can expose weaknesses, but not whether they can remove them; blocking a known exploit may leave variants open or disable functionality.Offensive success is comparatively self-certifying, whereas defensive success requires checking both security and retained behavior.Source: §2 · p. 2
- Defensive benchmarks can rely on public CVEs, generated code, diagnosis hints, or narrow weakness coverage, each limiting what auditing capability they measure.Some benchmarks focus on C/C++ memory-safety bugs and cover only one to four OWASP Top 10:2025 categories.Source: §2 · p. 2
- CWE-bench addresses these shortcuts with real repositories, no diagnosis given to agents, and coverage of all ten OWASP Top 10:2025 categories.Its evaluation also prevents Internet access and requires both security and functional verifiers to pass.Source: §2 · p. 3, Figure 1 · p. 3
- Automated grading can accept incorrect patches, reject correct ones, or be gamed; CWE-bench combines a programmatic score, agentic judges, manual review of development disagreements, and API-only access.The cited discussion notes that judges can agree with each other while disagreeing with execution-based checks.Source: §2 · p. 3
3 Benchmark Design
CWE-bench models repository auditing as an interactive task, constructs a broad private set of injected vulnerabilities, and validates its programmatic grading against agentic judges.
3.1 Problem formulation
- A rollout receives a binary reward on its final state, including after timeout, and passes only when both exploit blocking and existing functionality tests succeed.The state includes the repository, audit instruction, and interaction history; Internet-access attempts are blocked.Source: §3.1 · p. 3
3.2 Task coverage and construction
- The evaluation set contains 120 private held-out tasks spanning 73 CWE types, eight languages, and all ten OWASP Top 10:2025 categories.Each task starts from an open-source repository with a real CVE and adds private vulnerabilities without published fixes; Figure 2 breaks down tasks by language and OWASP category.Source: §3.2 · p. 3, Figure 2 · p. 4
3.3 Verifier validation
- Three agentic judges assess security, functionality, attack-variant robustness, grading integrity, and build soundness; disagreements with the verifier were manually reviewed during development.Reviews prompted probes for incomplete repairs, behavior-based checks in place of reference-patch structure assumptions, and corrections to judge rubrics.Source: §3.3 · p. 4
4 Experiments
Across eleven agents, the leaders tie at 68% pass@1, while retry outcomes, task coverage, and cost differ. Agent leadership also changes across languages and vulnerability categories.
4.1 Experimental Setup
- Each attempt runs in an isolated container with model-API access only; eleven agents make four independent attempts, each limited to one hour.Scores reflect each model together with its coding-agent harness; provider refusals count as failures.Source: §4.1 · p. 4
4.2 Main Results
- 81% pass@4 is Grok 4.7’s highest score; three agents tie at 68% pass@1, and the same four agents lead under both graders.The graders agree on 93.9% of individual attempts.Source: §4.2 · p. 4, Table 2 · p. 5
4.3 How do repeated attempts and different agents expand task coverage?
- Only 69 tasks are common to the three tied leaders, who together pass 110; Grok has 34 mixed-outcome tasks, versus GPT-6 Astra’s 15.Grok, Gemini, and GPT-6 Astra reach 81%, 75%, and 74% pass@4, respectively, from the same 68% pass@1.Source: Figure 3 · p. 5, §4.3 · p. 5
4.4 How does pass rate scale with inference cost?
- 68% pass@1 costs $2.75 for Grok, $2.85 for GPT-6 Astra, and $6.63 for Gemini; Claude reaches 67% at $0.79 on the Pareto frontier.These five agents form the frontier; moving from Claude to the 68% agents costs 3.5× more per rollout for one pass@1 point.Source: §4.4 · p. 5, Figure 4 · p. 6
4.5 How does performance vary across languages and vulnerability categories?
- Leaders trade places across languages; A08 is hardest, with a best score of 57%, while A04 is easiest, with Claude Opus 5.5 reaching 90%.Grok leads on C/C++, Gemini on Go, and GPT-6 Astra on JS/TS and Python.Source: Figure 5 · p. 6, §4.5 · p. 6
5 Discussion
Across eleven agents, equal pass@1 hides differences in retry gains, coverage, and cost, while vulnerability category matters more than language. The discussion connects these findings to routine repair, benchmark breadth, and verifier reliability.
5 Discussion
- Across eleven agents, leaders tie on pass@1 but differ in retries, coverage, and cost, while vulnerability category matters more than language.No single agent is best across languages or vulnerability categories.Source: §5 · p. 7
- Agents reach 81% pass@4 with retries, and together cover 110 tasks versus 97 for the best single agent, making multi-agent systems a natural direction.Claude Opus 5.5 comes within a point of the leaders at a fraction of their cost, which the paper says makes repair cheap enough for routine use.Source: §5 · p. 7
- CWE-bench covers 73 CWEs, but this is a small fraction of the more than 900 catalogued weaknesses, so broader category coverage remains important.The discussion argues that narrow benchmarks can rank agents differently and that per-class results reveal weaknesses hidden by aggregate scores.Source: §5 · p. 7
- The programmatic verifier and judge panel agree on 93.9% of attempts (κ = 0.88), and the same four agents lead under both.CWE-bench grades attempts with both a programmatic verifier and an independent panel of agentic judges.Source: §5 · p. 7
6 Conclusion
The conclusion contrasts offense-focused evaluations with the question of whether agents can make software safer. CWE-bench measures vulnerability discovery and repair, while the results show differences hidden by a single pass rate.
6 Conclusion
- Security evaluations have largely measured agents’ ability to attack software, while deployment in code auditing and maintenance makes defensive capability consequential.The conclusion frames the central question as whether agents make software safer.Source: §6 · p. 7
- CWE-bench tests whether agents can find and repair unlabeled vulnerabilities in real repositories, accepting patches only when they block exploits and preserve behavior.Source: §6 · p. 7
- Agents differ in retry response, task coverage, cost, and vulnerability classes handled, differences that a single pass rate hides.The conclusion calls for rigorous measurement across the breadth of vulnerabilities in real software as offensive capabilities improve.Source: §6 · p. 7
A Evaluation Details
The evaluation pairs each model with a harness and runs four independent attempts per task under a one-hour limit. It also specifies how timeouts, provider safety flags, isolation, and attempt costs are handled.
A Evaluation Details
- Each model-harness pair runs at high reasoning effort and makes four independent attempts per task, each limited to one hour.An attempt reaching the time limit is graded on the repository state at that point.Source: A Evaluation Details · p. 10
- Attempts run in isolated containers whose only network access is the model provider’s API, with provider-specific handling of flagged defensive requests.A flagged Codex request ends in failure, while Claude Code completes it using a fallback model in the same session; those attempts are credited to the original model.Source: A Evaluation Details · p. 10
- Attempt cost comes from the harness when reported, or is computed from token usage and the provider’s published per-token prices.Source: A Evaluation Details · p. 10
B Agentic Judge Panel
Three agentic judges assess patched repositories independently using the task instruction, vulnerability description, reference patch, and repository tools. A patch is accepted by the panel when a majority of judges accept it, with each judge requiring all five criteria.
B Agentic Judge Panel
- Each judge works independently in a copy of the patched repository and can inspect code, build the project, run tests, and run the programmatic verifier.Source: B Agentic Judge Panel · p. 10
- A judge accepts a patch only when all five criteria are satisfied: security, functionality, robustness to attack variants, grading integrity, and build soundness.Source: B Agentic Judge Panel · p. 10, Table 3 · p. 10
- The panel accepts a patch when a majority of judges accept it.Source: B Agentic Judge Panel · p. 10
C Agreement between Programmatic and Panel Verdicts
The programmatic verifier and judge panel agreed on most final-evaluation attempts, but often differed over whether patches blocked attack variants. During development, manual review informed revisions to the verifier’s behavior checks.
C Agreement between Programmatic and Panel Verdicts
- 200 of 320 disagreements were patches accepted by the verifier and rejected by the panel.Source: C Agreement between Programmatic and… · p. 10
- The verifier runs fixed exploit probes and functionality tests, while judges also assess whether patches block attack variants beyond those probes.Source: C Agreement between Programmatic and… · p. 10
- 93.9% of 5,280 final-evaluation attempts received matching programmatic and panel verdicts (Cohen’s κ = 0.88).Per-agent agreement ranged from 91.0% to 95.4%.Source: Table 4 · p. 10, C Agreement between Programmatic and… · p. 10
D Human Review
- During development, reviewers resolved disagreements in favor of the verifier unless it was shown to be wrong, prompting behavior checks for valid repairs and added probes for incomplete ones.Issues affecting neither security nor functionality did not override the verifier’s verdict.Source: D Human Review · pp. 10–11
E Task Coverage
Table 5 organizes the evaluation-set CWEs by OWASP Top 10:2025 category.
E Task Coverage
- The table groups those CWEs by category rather than presenting an ungrouped list.Source: E Task Coverage · p. 11, Table 5 · p. 11
- Table 5 lists evaluation-set CWEs under their OWASP Top 10:2025 category.Source: Table 5 · p. 11, E Task Coverage · p. 11
- The table’s scope is the CWEs included in the evaluation set.Source: Table 5 · p. 11
F Item Response Theory Analysis
The Rasch model places agents and tasks on a shared ability–difficulty scale. It estimates tiered success curves and identifies tasks that remain difficult even for the strongest agents.
F Item Response Theory Analysis
- The model fits passing attempts out of total attempts and places agent ability and task difficulty on the same scale.Abilities are centered at zero, with intervals from 300 bootstrap refits.Source: F Item Response Theory Analysis · p. 11
- 30 tasks have estimated difficulty above the strongest agent’s ability, so that agent is predicted to solve each less than half the time.The count ranges from 25 to 34 across bootstrap refits.Source: F Item Response Theory Analysis · p. 11, Figure 6 · p. 12
- On the median-difficulty task, predicted success is 78% for the top tier, 58% for the middle tier, and 28% for the lower tier.Source: F Item Response Theory Analysis · p. 11, Figure 6 · p. 12