Source-linked AI summary
How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark
arXiv:2610.38617v1cs.CRcs.AIcs.SE
TL;DR
The paper asks what AI security agents find beyond deterministic SAST and what SAST catches that agents miss. It benchmarks full scanner configurations and finds neither sufficient alone: Snyk Code plus one AI review matches 101 of 108, the ceiling across comparable configurations.
Problem
The study asks what AI security agents and harnesses find that deterministic SAST does not, and what SAST catches that they miss.
Source: §1 · p. 1Method
VulnBench 2.0 evaluates full scanner configurations using attacker-reachable vulnerability recall and endpoint-aware scoring.
Source: §1 · p. 2, §2.6 · p. 7Results
Snyk Code and one AI review together matched 101 of 108 vulnerabilities, the ceiling reached by all seven comparable configurations combined.
Source: §10 · pp. 29–30Takeaways & Limitations
The authors recommend running deterministic SAST and AI review together and evaluating an AI tool’s built-in policy and localization behavior.
Source: §10 · p. 30Only Snyk Code represents SAST, and the design cannot separate harness contributions from model contributions, limiting generalization to other analyzers and model-only effects.
Source: §7.2 · p. 26
Abstract
from arXiv · showhide
We ask what AI coding agents and AI security harnesses find that deterministic static application security testing (SAST) does not, and what they miss that SAST catches. VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java; 62 flow across more than one file. A finding is credited only if it locates the vulnerability's attacker-controlled source and its dangerous sink, each within two lines. We evaluate nine productized configurations under one blind protocol: the Snyk Code static analyzer, Claude Code's /security-review with three model and effort settings (one with the Snyk MCP server), Codex Security with GPT-5.6 Luna, Terra and Sol, and the DeepSec scanner with two model backends. Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart. Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined, and the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report. What an AI engine misses depends on its harness as much as on its model: without Claude Code's built-in review policy, the same model matched 16 instead of 9 of the 20 vulnerabilities in the classes that policy excludes or restricts (supplementary run on 19 of 20 applications); Codex Security with GPT-5.6 Sol located 80 vulnerabilities at file level but 34 at line level; DeepSec's model-free file selection kept a file from only 56 of the 108 flows. Coverage and cost were only loosely related. The harness, fixtures, reference sets and all run data are public.
1 Introduction
VulnBench 2.0 asks what productized AI security harnesses find beyond deterministic SAST and what SAST catches that they miss. It finds complementary coverage, while harness policy, localization, and cost shape AI configurations’ results.
Benchmark design
- The benchmark addresses limits of prior work with application-scale fixtures, attacker-reachable curated ground truth, endpoint-aware scoring, and productized security harnesses.A finding counts only when it locates the attacker-controlled source and dangerous sink within two lines; results are attributed to the complete harness and model configuration.Source: §1 · p. 2
Complementarity
- Together, Snyk Code and Claude Code with Opus 5 at xhigh matched 101 of 108 vulnerabilities, the ceiling reached by all seven comparable configurations.Snyk Code added 16 vulnerabilities to the agent and the agent added 12 to Snyk Code; the study says the 28 disagreements are too few to distinguish their standalone scores from chance.Source: §1 · p. 2
Additional findings
- The AI configurations reported at least seven real vulnerabilities outside the reference set that Snyk Code did not report.The authors note that findings outside the reference set are not necessarily false.Source: §1 · p. 2
Harness policy
- Without /security-review, Opus 5 at xhigh matched 16 of 20 vulnerabilities in classes that the policy excludes or restricts, versus 9 with the built-in policy.The supplementary prompt-swap run covered 19 of 20 applications.Source: §1 · p. 2
Efficiency and robustness
- Snyk Code scanned all 20 applications in 8 minutes, while comparable AI configurations took 2.4 to 8.9 hours; within Codex Security, Sol cost 35.0 times as much as Luna and matched 11 fewer vulnerabilities.The paper also reports that Snyk Code and Opus 5 xhigh remained the top two under all 20 alternative scoring policies tested.Source: §1 · p. 3
Contributions
- The paper’s stated contributions include the first endpoint-matched SAST–agent overlap measurement and evaluation of Claude Code, Codex Security, and DeepSec on curated endpoint-annotated ground truth.The evaluation also examines line offsets and recall ceilings imposed by harnesses before models run, and tests its harness-policy reading with a prompt-swap control.Source: §1 · p. 3
2 Benchmark and method
VulnBench 2.0 evaluates nine complete scanner configurations against a curated, attacker-reachable reference set across 20 applications. Its protocol is blind, its primary scoring requires endpoint localization, and supplementary runs test specific interpretations of the results.
2.1 Corpus
- The corpus contains 108 vulnerabilities across 20 applications, including 62 flows spanning multiple files and 447 authored flow locations.It covers four language families and 14 vulnerability classes; the median is five vulnerabilities per application.Source: §2.1 · pp. 3–4
2.2 Reference set
- Each reference vulnerability is author-curated as attacker-reachable, annotated with source-to-sink flow locations, and backed by an author-verified exploit proof of concept.The reference set is independent of tool findings, though one known defect is disclosed and its origins and seeding process may introduce bias.Source: §2.2 · p. 4
2.3 Configurations
- The nine evaluated configurations are complete harness-and-model combinations, including Snyk Code, Claude Code, Codex Security, and DeepSec.Results are attributed to the full configuration rather than to a model alone.Source: §2.3 · p. 4
2.4 Protocol
- All nine configurations ran once on each of 20 applications under blind, independent runs in fresh isolated workspaces, for 180 successful runs.No configuration received vulnerability descriptions, class hints, advisories, or the reference set.Source: §2.4 · p. 6
2.5 Scoring
- Endpoint-aware scoring requires a class match and source-and-sink localization within two lines; DeepSec instead receives localized-recall scoring because its exports do not label source and sink.DeepSec results are reported separately and are not ranked or combined with the seven endpoint-aware configurations; exact-line sensitivity analysis is also reported.Source: §2.5 · pp. 6–7
2.6 Metrics and statistics
- Macro Attacker-Reachable Vulnerability Recall is the primary metric, averaged across applications; uncertainty intervals are approximate and do not capture run-to-run variation.The paper also reports pooled matches, treats precision as secondary, and notes it is a lower bound when the reference set may omit real vulnerabilities.Source: §2.6 · p. 7
2.7 Supplementary runs
- Three supplementary arms probe MCP effects, the built-in review policy, and Snyk Code repeatability; their results remain separate from the main tables and rankings.For the policy control, the default audit prompt replaced /security-review and completed on 19 applications; outcome thresholds were set before per-class analysis.Source: §2.7 · pp. 7–8
3 Complementarity: SAST and AI review cover different ground
Snyk Code and Claude Code’s Opus 5 xhigh review cover different vulnerabilities, and together match the ceiling reached by all seven comparable configurations. Their relative coverage depends on flow shape and harness policy, while combining them brings triage costs.
3.1 Main recall results
- 82.4% (89 of 108) for Snyk Code and 79.3% (85 of 108) for Opus 5 xhigh were too close to distinguish on this corpus.They disagreed on 28 vulnerabilities, split 16 to 12 (McNemar p = 0.57).Source: §3.1 · p. 8, Table 4 · p. 8, Figure 1 · p. 9
3.2 The ceiling
- 101 of 108 vulnerabilities were matched by Snyk Code plus Opus 5 xhigh, equaling the union of all seven endpoint-aware configurations.After those two, no other configuration added a vulnerability; the best pair without SAST reached 89.Source: §3.2 · p. 9, Figure 2 · p. 10
3.3 What each engine brings
- Snyk Code added 16 vulnerabilities to Opus 5 xhigh, while Opus 5 xhigh added 12 to Snyk Code, seven of them cross-file.Snyk Code’s additions were mostly same-file; its strongest class coverage included all 11 open redirects, 6 prototype-pollution vulnerabilities, and 16 path traversals.Source: §3.3 · p. 11
3.4 Where each engine wins, by flow shape
- Snyk Code matched 40 of 46 same-file vulnerabilities versus Opus 5 xhigh’s 32, while cross-file results were 49 of 62 versus 53.Cross-file cross-site scripting was difficult for every configuration: none matched more than 4 of 7.Source: §3.4 · pp. 11–12, Figure 4 · p. 12
3.5 The cost of combining
- 539 findings from the combined engines matched 101 reference vulnerabilities, and Snyk Code’s precision was 69% in the 14 scored classes versus Opus 5 xhigh’s 81%.Most of Snyk Code’s unmatched findings were outside the scored classes; they are not thereby established as false positives.Source: §3.5 · p. 12, Figure 5 · p. 13
3.6 Findings outside the reference set
- At least seven real vulnerabilities in scored classes were found by AI configurations but absent from both the reference set and Snyk Code’s findings.They were live-reproduced; this is a lower-bound count from consensus candidates, not an estimate of any engine’s miss rate.Source: §3.6 · pp. 13–14
- All seven additional vulnerabilities crossed a boundary that first-party, single-language taint analysis typically does not follow.Six passed through third-party code and four crossed from server-side code into the browser’s document object model.Source: §3.6 · p. 14
- Codex Security configurations reported all seven additional findings, Claude Code configurations six, and DeepSec three, but the consensus filter favors families with more configurations.The reviewed candidates received AI-agent first and skeptical second passes, an author’s final call, and AI-assisted exploitability checks.Source: §3.6 · p. 14
4 Harness policy: the harness decides what gets reported
Harness instructions and pipeline choices shape which vulnerabilities reach reports, while model differences still matter within a fixed harness. The section shows distinct reporting, localization, and file-selection constraints across three harnesses.
4.1 Claude Code /security-review: a precision-first policy that suppresses real classes
- The built-in policy excludes ReDoS and restricts prototype pollution and open redirect; Snyk matched 18 of those 20 vulnerabilities, versus 9 for Opus 5 xhigh.On the 81 other vulnerabilities, Opus 5 xhigh matched 71 and Snyk Code 67; the per-class evidence varies in strength.Source: §4.1 · p. 15
- 16 of 20 policy-class vulnerabilities were matched with the default audit prompt, versus 9 with /security-review, in a single-run control on 19 applications.The same Opus 5 model and effort were used; the control also reported the right class for all 20, versus 12 with /security-review.Source: §4.1 · pp. 15–16
4.2 Codex Security: the right function, a line above the dangerous call
- 80 vulnerabilities were located at file level by Codex Security with GPT-5.6 Sol, but only 34 met the two-line rule; 39 of 41 non-source locations were above the sink.Reported locations were often enclosing function or route headers; the median offset was six lines above the sink.Source: §4.2 · pp. 16–17, Figure 7 · p. 17
4.3 DeepSec: a recall ceiling fixed before the model runs
- DeepSec’s model-free selection included a flow file for 56 of 108 vulnerabilities and a labeled sink file for 44 of 107; neither backend matched unselected flows.Selection varied by language: all 12 Java flows, 23 of 30 JavaScript/TypeScript, 11 of 32 Go, and 10 of 34 Python flows had a flow file selected.Source: §4.3 · p. 18, Figure 9 · p. 19
4.4 The counterweight: models still matter inside a harness
- Within the same harness, effort, and prompt, Opus 5 matched 14 vulnerabilities Sonnet 5 missed, while Sonnet matched 2 Opus missed; the policy-swap control also used fewer resources.On 19 applications, the control matched 93 of 105 versus 83 for /security-review, at 0.63 times the cost and 0.52 times the wall time; this was a single supplementary run.Source: §4.4 · p. 19
5 Efficiency: what coverage costs
Coverage and resource use were only loosely related in this study. The deterministic scan was much faster, while AI configurations varied in cost, coverage, and review volume.
5.1 Time
- Coverage and cost were only loosely related: the most expensive configurations were not the most effective, and Snyk Code scanned all applications in 8 minutes versus at least 2.4 hours for AI configurations.Wall time includes the deployed harness pipelines and shared gateway, not model speed alone.Source: §5 · p. 19, §5.1 · p. 20, Figure 10 · p. 21
5.2 Dollars and tokens
- Per matched vulnerability, the cheapest known-cost configurations ranged from $0.47 for Codex Security with GPT-5.6 Luna to $0.72 for Opus 5 medium plus MCP; GPT-5.6 Sol cost $21.92.Claude Code configurations used 0.35–1.03 million tokens per matched vulnerability and Codex Security 24–42 million; token accounting differs by provider.Source: §5.2 · p. 20, Figure 11 · p. 22
5.3 More spend did not reliably buy more coverage
- Within Codex Security at xhigh effort, GPT-5.6 Sol cost 35.0 times as much as GPT-5.6 Luna and matched 11 fewer vulnerabilities; within Claude Code, Opus 5 cost 4.2 times as much as Sonnet 5 and matched 12 more.The first comparison holds effort fixed across model tiers; the second holds effort and prompt fixed.Source: §5.3 · p. 22
5.4 The Snyk MCP configuration
- The Opus 5 medium-effort MCP configuration matched 80 vulnerabilities, all also matched at xhigh, at 0.35 times the cost and 0.43 times the wall time.Effort, MCP access, and an appended prompt instruction differed, so the comparison does not isolate grounding.Source: §5.4 · p. 23
5.5 Scaling with application size
- Wall time tracked application size most closely for Codex Security GPT-5.6 Terra (Spearman ρ = +0.78) and DeepSec (+0.68 and +0.71); the 20-application analysis is descriptive.Application sizes ranged from about 480 to 60,000 non-blank source lines.Source: §5.5 · p. 23
6 Robustness of the results
The leading configurations remain stable across alternative scoring rules, but middle rankings move with scoring details, especially location tolerance. Supplementary checks support some robustness conclusions while the small sample limits claims about close rankings.
6 Robustness of the results
- Snyk Code and Claude Code with Opus 5 xhigh ranked in the top two under all 20 alternative scoring rules; Snyk ranked first under 18.The two exceptions were the most lenient location rules, under which Opus 5 xhigh ranked first; their order should not be read as a result.Source: §6 · p. 23, Figure 12 · p. 24
- Rank agreement ranged from Kendall τ = 1.00 to 0.33, with middle-table reordering concentrated in Codex Security and Sonnet 5 xhigh; widening tolerance to ±10 lines added 37 Sol matches.The same widening added 28 Terra matches, 16 Luna matches, at most 3 for any Claude Code configuration, and none for Snyk Code.Source: §6 · pp. 23–24, Figure 12 · p. 24
- Snyk Code returned byte-identical findings and the same 89 matches in two supplementary repetitions, but 108 vulnerabilities across 20 applications cannot establish the order of closely matched configurations.Rescoring tests sensitivity to scoring rules, not statistical power.Source: §6 · pp. 24–25
7 Threats to validity
The authors identify opposing construction biases and limits in the corpus, scoring, and evaluated configurations; several threats remain unmeasured, and the results’ reach beyond this setup is untested.
7.1 Reference-set construction, independence and authorship
- Snyk Code informed location formatting and class names during curation, but the authors cannot verify that every seeded vulnerability followed instructions to preserve findings it missed.All authors are Snyk employees, and five applications had appeared in Snyk demonstrations, though none was used in Snyk Code development or rule testing.Source: §7.1 · pp. 25–26
- Seeding could favor either family: same-family models may recognize their own patterns, while examples from Snyk’s database may aid Snyk Code; the net effect is unknown.Seed-to-application assignments and prompt examples were not recorded; Codex Security’s class-level matches were 79 to 88 despite its low endpoint-aware scores.Source: §7.1 · p. 25
- Removing alternative class names changes one match, while adding two adjudicated cross-site scripting labels raises AI configurations by at most one match each.Labels outside Snyk Code’s vocabulary could go uncredited, but the measured effect on matching is small.Source: §7.1 · p. 25
- Without the five applications previously used in Snyk demonstrations, Snyk Code and Opus 5 xhigh remain the top two and their union still equals all seven configurations’ union.They match 74 and 72 of the remaining 87 vulnerabilities, respectively; their union is 83.Source: §7.1 · p. 25
7.2 Remaining threats
- Each configuration–application cell ran once, leaving AI run-to-run variation unmeasured; four runs were retried after failed attempts, and only final attempts were scored.Three failed attempts had been marked non-retryable by the harness.Source: §7.2 · p. 26
- Correcting one duplicated-location defect in the reference set raises the Opus 5 xhigh result from 85 to 86 and the all-configuration union from 101 to 102.The paper’s reported results use the frozen reference set, and no other configuration changes.Source: §7.2 · p. 26
- Reference-set omissions can move measured recall in either direction, while measured precision can only rise; the reference set also covers only 14 vulnerability classes.At least seven real omissions are confirmed, and reports outside the covered classes are out of scope regardless of validity.Source: §7.2 · p. 26
- Results concern full configurations, not isolated models: only Snyk Code represents SAST, and transfer to production code, other languages, or larger applications is untested.Scoring rules are choices; versions, run environment, costs, and statistical intervals also impose limits, including approximate intervals from only 20 applications.Source: §7.2 · pp. 26–27
8 Related work
Prior work compares static analyzers with language models, repository-scale benchmarks, and agent harnesses, but differs in scale, scoring, or the systems tested. This study directly evaluates packaged security configurations with endpoint-aware scoring.
Static analysis, language models and their combination
- Prior comparisons find language models detect more vulnerabilities with more false positives, while combining models and SAST can mitigate weaknesses of each [27]; line-level misplacement motivates hybrid pipelines [11].Studies also find that combining static analyzers reduces misses at the cost of more functions flagged, and investigate related detector combinations.Source: §8 · p. 27
- This study measures complementarity between one deterministic SAST engine and deployed AI security tools, attributing coverage differences to vulnerability classes and flow shapes.CASTLE, RealVuln, IRIS, and Revelio instead compare scanners or combine language models with static-analysis components [7] [21] [17] [13].Source: §8 · p. 28
Repository-scale benchmarks with located ground truth
- Repository-scale benchmarks annotate vulnerability locations, but VLoc Bench gives weakness classes and scores returned files, whereas this study gives no vulnerability information and requires source-and-sink locations within two lines.Other benchmarks include VulnGym [14], X.js [1], HoF-Bench [23], and the predecessor VulnBench JS 1.0 [24].Source: §8 · p. 28
The harness as part of the result
- Prior harness studies report mixed controlled-swap results, while SecPRBench found Codex Security lowered recall by 8.3 points relative to the same model, mostly through abstention [25].Other work reports differing subsets across cybersecurity scaffolds, and no resolvable average advantage between native and neutral coding harnesses [4].Source: §8 · p. 28
9 Future work
The authors propose extending the reference set, repeating AI runs and policy controls, and testing direct engine combinations. They also call for better coverage of cross-layer flows and more reproducible run records.
Cross-layer rendering flows
- A dedicated corpus of multi-layer rendering flows could measure cross-file XSS that spans back-end and front-end frameworks, where every engine currently struggles.No endpoint-aware configuration matched more than four of seven such vulnerabilities; seven cases are too few for stronger conclusions.Source: §9 · p. 29
A revised reference set
- A revised reference set should add seven adjudicated vulnerabilities, two missing XSS class names, the corrected iguana entry, and adjudications of singly reported findings.Results on the revised set should be reported as a new version rather than mixed with current numbers.Source: §9 · p. 29
Repetitions
- Repeating leading AI configurations would estimate run-to-run intervals for recall, cost, and time and test whether their complementarity is stable.Snyk Code’s output was identical across three runs, but each AI configuration ran once per application.Source: §9 · p. 29
Separating policy from model
- Repeating prompt swaps across Claude Code configurations would test whether the review-policy effect generalizes across models and varies by vulnerability class.The existing control covered one configuration once.Source: §9 · p. 29
New ways to combine the engines
- Direct combination experiments could give agents SAST coverage information, iteratively target earlier misses, or use AI to triage SAST findings.Apart from one Snyk MCP configuration receiving Snyk findings, engines ran independently and were combined afterward.Source: §9 · p. 29
Protocol
- Future runs should record every tool version per run and pin MCP server versions to avoid bounded exact re-execution.The current archive lacks per-run Snyk CLI versions and used an unpinned Snyk MCP server.Source: §9 · p. 29
10 Conclusion
The study compares deterministic SAST and AI security configurations on attacker-reachable vulnerabilities using source-and-sink localization. Neither alone covers the benchmark: together they reach the combined ceiling, while AI misses depend substantially on harness behavior.
Conclusion
- The evaluation covers 108 attacker-reachable vulnerabilities and credits findings only when they locate both the vulnerable flow’s source and sink.Both endpoints must be within two lines.Source: §10 · p. 29
- Snyk Code and Claude Code /security-review with Opus 5 xhigh together match 101 of 108 vulnerabilities, the ceiling of all seven comparable configurations combined.Snyk Code contributes 10 vulnerabilities no AI configuration matched; Opus 5 xhigh adds 12 that Snyk Code missed, including seven cross-file cases.Source: §10 · pp. 29–30
- AI misses depend heavily on harness policy, localization, and file selection, while a more expensive model did not reliably buy more coverage.Snyk Code scanned all applications in eight minutes; every AI configuration took hours.Source: §10 · p. 30
- Practitioners are advised to run deterministic SAST alongside AI review and consider built-in policy and localization behavior; the benchmark and run data are public.The authors also recommend scoring complete configurations and requiring line-level evidence.Source: §10 · p. 30
Data and code availability
The benchmark materials and run data are public, enabling reproduction and re-scoring. The release includes the harness, fixtures, reference sets, datasets, and finding reproducers.
Data and code availability
- The public repository contains the harness, scorer, 20 fixtures, reference sets, scoring documentation, and the main dataset archive.The archive includes a SHA-256 checksum, per-file checksums, its parent execution record, and extraction instructions.Source: Data and code availability · p. 30
- Public supplementary datasets and a post-disclosure Snyk Code re-scan are archived separately; the re-scan was never merged into the main results.Reproducers for findings outside the reference set are also public, one directory per finding.Source: Data and code availability · p. 30
- The companion website is vulnbench.com, and VulnBench 2.0 identifies VulnBench JS 1.0 [24] as its predecessor.The paper states that everything needed to reproduce or re-score it is public.Source: Data and code availability · p. 30
- Re-running the deterministic scorer on archived findings reproduces every paper score, and the analysis re-derived all 180 run scores exactly.The harness and benchmark data are released under Apache License 2.0; fixture applications retain their upstream licenses.Source: Data and code availability · p. 30
A Corpus and dataset details
The appendix documents the corpus, run matrix, prompts, versions, and scoring so readers can reproduce or challenge reported rows. The reference set uses distinct raw and scoring location counts.
A Corpus and dataset details
- The raw reference basis counts 447 locations, while scorer-merged locations total 443, averaging 4.10 per vulnerability.Raw locations include 131 sources, 118 sinks, and 198 unlabeled intermediate steps; flows span 1–19 locations.Source: A Corpus and dataset details · p. 34
- The 108 vulnerabilities span 14 classes; 90 require matched source and sink, 15 accept either endpoint, and 3 have one location.Twenty-one vulnerabilities list multiple acceptable sources or sinks; five are recorded as crossing a service boundary, a field the scorer does not use.Source: A Corpus and dataset details · p. 34, Figure 13 · p. 34, Figure 14 · p. 35
- The fixture lookup covers 20 applications, including four written by Snyk employees for earlier work; one iguana entry has a reference-set defect.The lookup defines cross-file, same-file, service-boundary, source/sink, and raw-location columns.Source: A Corpus and dataset details · p. 35, Table 5 · p. 35
B.1 Executions and harness source
- The main matrix comprises 180 successful runs from early September 2026, with 140 Snyk Code, Claude Code, and Codex runs and 40 DeepSec runs.Runs started in fresh isolated workspaces containing only application source; model traffic used one LiteLLM gateway.Source: §B.1 · p. 36
B.2 Versions
- Snyk CLI 1.1305.1 is operator-reported for the main runs, while the Snyk MCP server version is inferred as 1.1307.0 from npm publish dates.The main archive does not record the CLI version, and the MCP version was not observed directly.Source: §B.2 · p. 36
B.3 Configuration parameters
- Recorded Claude turns can exceed the 30-turn setting because they include sub-agent messages; DeepSec turns are summed over batches.Run parameters are tabulated from the main execution’s configuration snapshot.Source: §B.3 · p. 37, Table 6 · p. 37
B.4 Prompt delivery
- Claude’s benchmark prompt requests complete attacker-reachable source-to-sink flows, while Codex Security and DeepSec use built-in instructions and Snyk Code has no prompt.Claude’s MCP configuration appends an instruction to invoke Snyk Code once; arm B instead uses the benchmark prompts without /security-review.Source: §B.4 · pp. 37–38
B.5 The built-in review policy
- Claude Code’s built-in review policy excludes denial-of-service findings, restricts prototype-pollution and open-redirect reports to extremely high confidence, and treats many framework XSS cases as safe.React and Angular XSS are treated as safe unless an unsafe rendering API is used; the policy block is identical across Claude binaries in the harness.Source: §B.5 · p. 38
C.1 Findings outside the reference set
The appendix separates seven reproduced AI-reported vulnerabilities outside the reference set from benchmark scores, then examines credit policy, equal-effort runs, runtime, and localization evidence.
C.1 Findings outside the reference set
- Seven real, new vulnerabilities reported by AI configurations but not Snyk Code were reproduced live and do not change benchmark scores.Table 7 lists the findings and reporting configurations; public reproducers are available.Source: §C.1 · p. 39, Table 7 · p. 39
C.2 Credit policy
- Under alternative many-to-one credit, Snyk Code’s counts stay unchanged while both Opus 5 configurations gain the most.The figure compares one-to-one matched vulnerabilities with many-to-one credit and also reports fully matched same-class groups within applications.Source: Figure 16 · p. 40
C.3 Equal-effort comparison for the Snyk MCP configuration
- At equal medium effort, Claude Code with and without Snyk MCP matched 80 and 81 of 108 vulnerabilities, respectively, and disagreed on five, split 2 to 3.The figure cautions that cost, time, and noise differences between the two single runs about a month apart are not effects of grounding.Source: Figure 17 · p. 41
C.4 Wall time and application size
- Wall time per application is plotted against approximate application size on log scales for seven endpoint-aware configurations; the association is descriptive.The comparison uses 20 applications and approximate non-blank source-line counts.Source: Figure 18 · p. 41
C.5 Localization evidence
- Codex Security most often reported a source without its sink, or the right file at other lines, among its best location evidence.The figure categorizes reported locations within two lines for findings of the right class; categories are not one-to-one consumed and can exceed matched totals.Source: Figure 19 · p. 42
C.6 Recall per application
Application-level recall varies across Snyk Code and the best Claude Code configuration, while the appendix presents per-application and per-class matched counts.
C.6 Recall per application
- Snyk Code recall exceeded the best Claude Code configuration on 8 applications, fell below it on 4, and tied on 8.Application rankings were nearly identical for the two Opus 5 configurations (Spearman ρ = +0.92); Opus 5 xhigh versus Snyk Code had ρ = +0.35.Source: §C.6 · p. 42, Figure 20 · p. 43
- Sonnet 5 xhigh’s application ranking agreed equally with Opus 5 xhigh and Snyk Code, with Spearman ρ = +0.64 for each comparison.Source: §C.6 · p. 42
C.7 Recall per vulnerability class
- The per-class figure reports matched vulnerability counts, and cells for classes with fewer than five vulnerabilities should be read as counts, not rates.Source: Figure 21 · p. 44