Source-linked AI summary
Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang, Zhiheng Lyu, Lingming Zhang
TL;DR
Existing benchmarks often derive reviewed changes from later fixes or fail to assess whether an agent explains a vulnerability’s root cause in a real introducing change. SecPRBench builds and validates paired real-world changes and evaluates findings against target-specific criteria; across eight models, recall ranges from 32.5% to 78.1%, with most misses involving misdiagnosis rather than simply overlooking the flawed file.
Problem
Existing benchmarks often use later fixes or do not measure whether agents diagnose vulnerabilities in real bug-introducing changes.
Source: §1 · p. 1Method
SecPRBench pairs real vulnerability-introducing changes with repaired companions and uses an LLM judge to check root-cause explanations against the reviewed code.
Source: §1 · p. 1, §4.1 · p. 5Results
Takeaways & Limitations
Better diagnosis is the key remaining challenge, and the released tasks, reports, and verdicts let others inspect and re-grade findings.
Source: §6 · p. 9The benchmark covers publicly disclosed vulnerabilities in nine widely deployed C, C++, and Rust projects, so its conclusions concern systems-code review and do not measure how often ordinary pull requests are vulnerable or report precision.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.
1 Introduction
SecPRBench evaluates whether agents diagnose vulnerabilities in real bug-introducing changes, addressing benchmarks that often use later fixes or score detection without testing the explanation. Across eight models, results expose both substantial performance differences and frequent misdiagnoses.
Motivation
- Existing benchmarks often score binary labels or derive vulnerable changes from later fixes, rather than testing diagnosis on real vulnerability-introducing commits.Binary labels can be correct for the wrong reasons, and identifying a target without its flaw does not establish a diagnosis.Source: §1 · p. 1
Contribution
- SecPRBench provides 265 buggy tasks from nine widely deployed projects, each paired with a fixed companion, and credits findings only when they explain the target root cause still present in reviewed code.The projects include Firefox, V8, and OpenSSL; the benchmark targets systems with millions of lines of code.Source: §1 · pp. 1–2
- An automated agentic pipeline traces vulnerabilities to introducing changes, composes later fixes, and validates pairs mostly through PoC replay, with human adjudication for remaining cases.An LLM judge derives target-specific criteria from public security records and checks findings against those criteria and the reviewed code.Source: §1 · p. 2
Results
- In 68% of misses, agents report on a file holding the flaw but argue for a different mechanism; four models find about three in four targets and are statistically tied overall.Even combined, the eight models leave 23 targets unfound.Source: §1 · p. 2
- Codex Security lowers recall by 8.3 points, mostly by abstaining, while Defending Code raises recall by 7.7 points on completed tasks at higher cost and report volume.On repaired changes, the judge never rules that the removed target is still present.Source: §1 · p. 2
2 Related Work
Prior work studies vulnerability detection, code review, security-agent tasks, and judging free-form outputs, but these approaches do not jointly evaluate diagnosis on complete vulnerability-introducing changes. SecPRBench instead scores root-cause explanations against the reviewed code.
Reviewing code changes
- Unlike MalPR-Bench, which derives most vulnerable PRs from later fixes, SecPRBench presents the original change that introduced each vulnerability and was merged before disclosure.Code-review benchmarks generally use review comments across all concerns as ground truth rather than security-specific diagnosis.Source: §2 · p. 3
Vulnerability detection
- Function-level vulnerability labels can be ill-posed because flaws may depend on calling context, while SecPRBench scores root-cause explanations for whole changes with repository access and paired controls.PrimeVul reports that detection drops sharply under realistic evaluation; JitVul pairs functions from introducing and fixing commits, and VulnGym evaluates repository-wide audits.Source: §2 · p. 3
Security agent benchmarks
- Security-agent benchmarks score artifacts such as PoCs, patches, or exploits on whole systems, rather than explanations of code changes.They cover capture-the-flag, bug-bounty, reproduction, exploitation, and hint-free discovery tasks.Source: §2 · p. 3
LLM judges
- Because execution cannot score an explanation, SecPRBench’s judge derives criteria from a reference and checks both the target mechanism and whether it remains in reviewed code.Prior free-form-output judges use per-instance rubrics, while code judges can be swayed by response order, names, and comments.Source: §2 · pp. 3–4
3 Benchmark Design and Construction
SecPRBench constructs paired review tasks around validated vulnerability-introducing changes and their repairs. Agents see repository context, the diff, and commit text, while security evidence and labels remain with evaluators.
3.1 Task Formulation
- Each case yields a buggy task from the introducing change and a fixed companion in which the target is repaired.Partial-fix cases use a security-relevant change that failed to remove the target.Source: §3.1 · p. 4
- Agents receive a repository snapshot, diff, and commit-message text, but not review discussion, security records, fixing commits, validation evidence, or task labels.They may read code in the snapshot; tasks run in isolated containers without Git history and with network access limited to the model provider.Source: §3.1 · p. 4
- A finding detects the target only if it explains the reference’s causal failure and that failure is present in the reviewed code; fixed companions control only for their target.The collection selects concrete code-level failures with a plausible trust boundary or untrusted-input path and a causally connected fix; review diffs are limited to 30 files and 2,000 added plus deleted lines.Source: §3.1 · p. 4, §3.2 · p. 4
3.3 Task Label Validation
The benchmark validates that introducing changes make target vulnerabilities reachable and repairs remove them, combining mechanical checks with human adjudication. Its 265 cases span projects, fix dates, and time from introduction to repair.
3.3 Task Label Validation
- 217 cases pass mechanical checks and 48 pass human adjudication after structural checks.Mechanical validation mainly uses PoC replay on the parent, introducing, and repaired trees.Source: §3.3 · p. 5
- PoC validation requires the target failure to be absent on the parent, present on the buggy tree, and absent on the repaired tree; adjudicators resolve cases replay cannot settle using source and execution evidence.Seven partial-fix cases are retained only when source evidence links the remaining target to the attempted repair in changed code.Source: §3.3 · p. 5
3.4 Dataset Characteristics
- Figure 3 summarizes the 265 cases by project, fixing date, and time from introduction to fix.Flaws reported and fixed separately remain separate cases even when they trace to the same change.Source: §3.4 · p. 5
4 Protocol Grading
SecPRBench grades whether findings explain the target vulnerability’s live root cause, using a criteria-first judge and metrics that distinguish detection from ranking and reporting effort. The judges show strong agreement, while each metric captures a different aspect of review performance.
4.1 What Counts as a Detection
- A detection must explain the target’s causal failure and show that it remains live in the reviewed code; a similar finding or correct location alone earns no credit.The explanation must identify what triggers the failure, the unsafe operation, and the violated security condition; an exploit is not required.Source: §4.1 · p. 5
4.2 Criteria-First Judge
- The judge derives target-specific criteria, matches a finding’s mechanism against them, then checks whether the vulnerability remains present in the reviewed change.It checks for intervening repairs, validation, bounds checks, or synchronization that break the causal chain.Source: §4.2 · pp. 5–6
4.3 Judge Reliability
- 96.9% of 2,143 nonempty buggy reports received matching verdicts from Opus 5.5 and Sonnet 4.6; manual checks supported Opus on most audited disagreements.The diagnostic sample does not estimate overall judge accuracy.Source: §4.3 · p. 6
4.4 Metrics
- Target recall counts credited detections across buggy tasks, while Hit@1 measures whether the first finding detects the target.A task is missed if the agent returns nothing or its run errors; report volume and cost are also tracked, with construction and judging excluded from cost.Source: §4.4 · p. 6
5 Evaluation
Across 265 buggy tasks, leading agents find about three in four targets, but contested flaws remain difficult and most misses are misdiagnoses. Security-focused harnesses shift recall in different directions, and fixed companions serve as controls for the target flaw rather than proof that a change is safe.
5 Evaluation
- The evaluation compares eight agents on the same inputs, withholding the vulnerability reference and later fix; direct review uses general-purpose coding harnesses.It also tests GPT-5.6 sol in Codex Security and Opus 5.5 in Defending Code, with web tools disabled.Source: §5 · pp. 6–7, Table 2 · p. 7
5.1 RQ1: How Often Do Agents Find the Introduced Vulnerability?
- 78.1% was Grok 4.6’s recall, while Grok 4.6, GPT-6 sol, GLM 5.3, and Opus 5.5 were statistically tied at the top; all eight agents left 23 targets unfound.The leading agents found about half of contested targets, and the other four found at most one in five; the three leaders together found about nine in ten.Source: §5.1 · p. 7, Table 2 · p. 7
5.2 RQ2: Why Do Agents Miss?
- 68% of scanned direct-review misses included a finding in a root-cause file that the judge rejected as describing a neighboring mechanism.Logic errors had about half the average recall of memory-safety flaws, and recall fell from about seven in ten with one root-cause file to under four in ten with five or more; these splits are observational.Source: §5.2 · pp. 7–8, Figure 4 · p. 8
5.3 RQ3: Does a Security-Focused Harness Help?
- Codex Security lowered GPT-5.6 sol’s recall by 8.3 points, mostly through abstention, while Defending Code raised Opus 5.5’s recall by 7.7 points at higher cost.Defending Code found all ten concurrency targets, while no direct-review agent found more than six; the comparisons are descriptive because prompts, budgets, and operating details also differ.Source: §5.3 · p. 8
5.4 RQ4: How Do Agents Perform on Fixed Versions?
- No fixed companion received a positive target verdict, but this narrow check measures neither false-positive rates nor report precision.A fixed companion controls only for its target; location overlap alone would credit about three in four companion scans.Source: §5.4 · p. 8
6 Conclusion
SecPRBench evaluates vulnerability diagnosis on real code changes by pairing buggy tasks with repaired companions. The results show that agents still miss many targets, often through misdiagnosis, making better diagnosis the central remaining challenge.
6 Conclusion
- Across eight direct-review agents, recall ranged from 32.5% to 78.1%, and even the combined agents left 23 targets unfound.Most misses involved a finding in the flawed file that argued for a different mechanism.Source: §6 · p. 8
- The automated pipeline supports refreshes from new disclosures, and released tasks, reports, and verdicts let others inspect and re-grade findings.The authors identify better diagnosis as the key remaining challenge.Source: §6 · p. 9
Limitations
SecPRBench’s conclusions are bounded by its selected vulnerability population, validation and control design, and evaluation setup. These limits qualify what its recall and harness comparisons establish.
Limitations
- The benchmark covers publicly disclosed vulnerabilities in nine C, C++, and Rust systems, so its conclusions concern systems-code review rather than undiscovered flaws or ordinary pull-request vulnerability rates.Its targets were later disclosed and fixed, and every buggy task contains a validated target.Source: Limitations · p. 9
- 48 of 265 cases rely on human adjudication, and validation evidence varies across cases; the judge is also an evaluated model and Sonnet drafted most references.The authors report 96.9% agreement across 2,143 nonempty buggy reports and a manual audit, but the audit does not establish overall judge accuracy.Source: Limitations · p. 9
- Each agent has one run per task without matched token budgets, and results describe a model in its harness rather than the model alone.Harness comparisons fix backend model and reasoning effort, but not prompts or budgets.Source: Limitations · p. 9
- Public security records and fixes may occur in model training data, although agents review without the security record, fixing commit, Git history, or web tools.For five models with fixes on both sides of documented cutoffs, adjusted recall was lower after the cutoff for four, with no statistically clear difference.Source: Limitations · p. 9
Ethics Statement
The paper describes its ethical safeguards and how it collects, selects, and validates benchmark tasks. Construction combines public security evidence with mechanical checks and human adjudication where needed.
Ethics Statement
- The benchmark uses public security records and fixes to validate historical vulnerabilities, reuses or generates PoCs only for label validation, and collects no personal data beyond published developer names.The authors state that benchmark labels do not establish a new vulnerability in a current release; new findings require independent validation and responsible disclosure.Source: Ethics Statement · p. 9
- An AI assistant and language-model agents helped with artifact inspection, task construction, PoC preparation, evidence interpretation, scoring, and manuscript editing; the authors retain responsibility for data and claims.Agent validation reports are not counted as independent human judgments.Source: Ethics Statement · p. 9
A Collector Inputs and Inclusion Evidence
- Collectors use CVE-linked records and public project security reports, including non-CVE issues, and retain cases with concrete security evidence and traceable introducing and fixing revisions.Documentation-only, test-only, dependency-update, and build-metadata changes are excluded; revisions must be on mainline, with fixes committed from January 2025 through June 2026.Source: A Collector Inputs and Inclusion… · p. 12
- All 265 selected buggy changes and fixed companions meet caps of 30 changed files and 2,000 added plus deleted lines, with no minimum-line threshold.The retained introducing and fixing revisions and source evidence support independent eligibility checks.Source: A Collector Inputs and Inclusion… · p. 12
B Validation Evidence and Adjudication
- Of 265 cases, 217 pass mechanical checks and 48 require human adjudication; partial-fix cases are retained only when source evidence links the remaining target to changed logic.The companion must remove the target, and unrelated pre-existing flaws do not qualify.Source: B Validation Evidence and Adjudication · p. 12
- Mechanical validation reconstructs parent, buggy, and repaired snapshots, requiring the same PoC and invocation to trigger the target only on the buggy tree.The observed failure must match the reference root cause; unrelated crashes, build failures, and timeouts do not establish a positive verdict.Source: B Validation Evidence and Adjudication · p. 12
C Task Schema and Snapshot Contract
The task schema separates agent-visible review inputs from evaluator-only references and validation metadata. The prompts direct reviewers to investigate changed code and a criteria-first judge to assess whether findings identify the target mechanism.
C Task Schema and Snapshot Contract
- Construction agents identify the root cause and revisions, then package the introducing change and compose a fixed companion; the loader withholds references and evaluator-only metadata from reviewers.When mechanical repair composition fails, a construction agent adapts the repair to the older source.Source: C Task Schema and Snapshot Contract · pp. 13–14
- The shared validator reconstructs buggy and repaired snapshots from a common parent, while agents review companions at the recorded fixing revision.A structural check establishes patch applicability, not that the target was removed.Source: C Task Schema and Snapshot Contract · p. 14
D Prompt Design and Summaries
- Claude Code review prompts provide the commit message and file diffs, tools for repository investigation, and instructions to trace relevant code assumptions and report concrete security failures.The Codex CLI prompt requests issues introduced or made reachable by the change and limits findings to changed files.Source: D Prompt Design and Summaries · pp. 14–15
- The judge derives task-specific criteria from the reference, then credits a finding only when the change evidence supports the target causal chain and no repair or mitigation breaks it.Criteria distinguish the target from neighboring bugs, location-only matches, hypothetical risks, and already-fixed failures.Source: D Prompt Design and Summaries · p. 15
E Scoring Details and Their Limits
The paper details its judging procedure and its limits, then reports run accounting, configuration, and paired comparisons. Results show that agents separate most on contested targets, while the leading agents are not statistically distinguishable overall.
E Scoring Details and Their Limits
- The judge returns structured verdicts after evaluating reports against criteria derived from the reference and relevant diff excerpts; empty reports count as non-detections without a judge call.Excerpts longer than 30,000 characters are truncated.Source: E Scoring Details and Their Limits · p. 15
- Manual adjudication covered 50 records, but disagreements were oversampled, so pooled audit percentages do not estimate overall judge accuracy.Agreement concerns binary verdicts, not every supporting claim; one correct negative relied on decisive code outside the judge excerpt.Source: E Scoring Details and Their Limits · p. 15
- The audit distinguishes mechanism matches from nearby failures: judges accepted a curl report describing the missing reset despite wording differences, but rejected an OpenSSL report with a different trigger and invariant.The audit does not validate neighboring findings or the curl report’s stronger memory-lifetime claim.Source: E Scoring Details and Their Limits · pp. 15–16
F.1 Run Accounting and Configuration
- All direct-review agents and Codex Security completed every pair, while Defending Code completed 260 of 265; run artifacts supply usage measures, and nominal dollar estimates exclude judging and construction.Runs had no turn or token budget and timed out after 60 minutes; output-token comparisons are only within a backend.Source: F Additional Results · p. 16, §F.1 · p. 16
F.2 Paired Comparisons
- Exact two-sided McNemar tests, counting missing or errored runs as misses, found no significant differences among Grok, GPT-6 sol, GLM, and Opus, even before correction for multiple comparisons.After Holm adjustment across all 28 direct-review pairs, no difference was significant within the top four agents or among Sonnet, GPT-5.6 sol, and GPT-6 luna; all remaining pairs had p < 0.001.Source: §F.2 · p. 16
- On contested targets, the top four agents find 48–57%, compared with 10–19% for Sonnet, GPT-5.6 sol, and GPT-6 luna and 5% for MiniMax M3.On easy targets, every agent except MiniMax M3 finds at least 90%.Source: §F.2 · p. 16
F.3 Results by Task Characteristics
Detection varies with bug class and task structure, and many misses involve a plausible but incorrect diagnosis rather than silence. Results on fixed companions, harnesses, judge outputs, and temporal comparisons reveal important evaluation boundaries.
F.3 Results by Task Characteristics
- Across the eight direct-review agents, recall was 51.5% for logic errors and 73.1% for memory-safety flaws.Logic errors accounted for 13 of 23 unfound targets; recall was lower for changes touching more than five files, though that difference was not statistically clear for two agents, and age comparisons were inconclusive.Source: §F.3 · pp. 16–17
F.4 Detection Failures
- About 68% of pooled direct-review misses were filed in root-cause files but rejected for the mechanism; recall was 69.6% with one root-cause file versus 38.5% with five or more.The root-cause-file split is observational and includes annotated supporting files; even all eight agents left 23 targets unfound.Source: §F.4 · p. 17, Table 9 · p. 18
F.5 Report Order
- The credited finding ranked first in 59.2% of multi-finding detections with resolved ranks, compared with 33.1% under random ordering.The judge sees findings in reported order, so a judge-position effect cannot be excluded.Source: §F.5 · pp. 17–18
F.6 Fixed Companions
- 92.3% of nonempty companion reports placed a finding in a root-cause file; location-only credit would accept 74.9% of companion scans versus 63.3% of buggy scans.After a category-D miss, 88.6% of companion reports still had a finding in a root-cause file.Source: §F.6 · p. 18
F.7 Review Harnesses
- On 260 completed tasks, Defending Code recall was 81.5% versus direct-review Opus’s 73.8%; Codex Security cost 5.1 times direct-review GPT-5.6 sol.Defending Code filed more findings at higher cost, and the comparison is descriptive because prompts and budgets differ; Codex Security found nine targets missed by direct review but missed 21 found only by it.Source: §F.7 · p. 19
F.8 Judge Output Fields
- Opus 5.5 and Sonnet 4.6 differed on 67 buggy-report verdicts, while agreement across 2,143 nonempty buggy reports was 96.9%.The sensitivity comparison excludes the Anthropic harness because its judge evaluations cover different task populations.Source: §F.8 · p. 19
F.9 Temporal Comparisons
- All five within-model cutoff comparisons were inconclusive, with intervals including zero; fixing dates do not establish training exposure or absence of contamination.Adjusted comparisons retain shared project–bug-class groups and use fixed pooled weights; sparse groups limit precision.Source: §F.9 · pp. 19–20, Table 12 · p. 20
G Artifacts and Reproducibility
- The release includes task metadata, results, a validation ledger, and code to rebuild tables without model calls, but rebuilding tables does not re-check judge verdicts.Replaying validation requires source trees, builds, and PoCs; reconstructing model outputs also requires model access, original prompts, and harness artifacts.Source: G Artifacts and Reproducibility · p. 20