Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

1. Introduction

• Language-model agents increasingly handle software-engineering tasks over real codebases, with these capabilities extending to security analysis.Source: §1 · p. 1
• A task pairs one GitHub Security Advisory with a weakness class and vulnerable and patched repository snapshots.Source: §1 · p. 2
• The task prompt supplies only a CWE description, illustrated with CWE-918, Server-Side Request Forgery.Source: §1 · p. 2

PHASE A · LOCALIZATION PHASE B · VERIFICATION

• Existing security evaluations often select code in advance or score downstream repair or reproduction, leaving repository-level localization itself unevaluated.Source: PHASE A · LOCALIZATION PHASE B ·… · pp. 2–3
• VLoc Bench pairs each vulnerable repository with its patched counterpart, gives agents only the CWE description and read-only access, and scores localization against patch-touched files.Source: Figure 1 · p. 2, PHASE A · LOCALIZATION PHASE B ·… · p. 3
• 0.229 File F1 is the strongest system’s score, while 38.4% of tasks receive no correct localization from any evaluated model.Source: PHASE A · LOCALIZATION PHASE B ·… · p. 3
• Strong Phase A localization does not guarantee that systems refrain from reporting files after remediation, so the benchmark evaluates both decisions.Source: PHASE A · LOCALIZATION PHASE B ·… · p. 3

Repository-Scale Localization in Software Engineering

• Software-engineering localization commonly ranks repository files against a reported defect, but typically assumes the issue is already known and may provide repository-specific evidence [5, 6, 19, 23, 44, 47, 48, 49].Source: §2 · pp. 3–4

Localization Is Implicit in Security Evaluation

• Security detection often evaluates preselected code, while repair or reproduction benchmarks may supply the location or judge execution instead of scoring where agents searched [27] [45], [3] [34].Source: §2 · p. 4

Vulnerability Localization as Repository Search

• Prior vulnerability-localization evaluations vary in target granularity and how much search remains, from functions or data-flow paths to whole repositories and supplied fixing commits [18] [26] [50], [18] [25] [26] [50].Source: §2 · p. 4

Localization When the Vulnerability May Be Absent

• Patched-code evaluation makes restraint part of localization: systems must judge whether evidence supports reporting any location, not only where a weakness lies [37] [45] [46] [50].Source: §2 · p. 4

Related work comparison

• To the authors’ knowledge, no prior benchmark scores file-level localization over a whole repository with the location withheld while also evaluating the corresponding patched repository.Source: §2 · p. 5

3. Benchmark Design

• VLoc Bench uses CWE descriptions to specify a weakness without repository-specific localization cues, enabling a common task across heterogeneous repositories and ecosystems.Source: §3 · p. 6

3.1. Benchmark Construction

• The benchmark targets patch sites: implementation files modified by the security fix, excluding test-only files, provide reproducible labels for defender investigation and remediation.Source: §3.1 · p. 6
• Each task pairs the commit immediately before a security fix with the patch commit, preserving repository structure so agents localize before the fix and verify after it.Source: §3.1 · p. 6

3.2. Evaluation Protocol

• Agents receive only a CWE description and read-only terminal access under shared prompts, tools, and command budgets; patched repositories should receive no vulnerability-file submission.Source: §3.2 · p. 7

3.3. Scoring

• Phase A uses File F1 to penalize both omitted and unrelated files, while Phase B uses TNR for clean-repository judgments; abstentions are reported separately.Source: §3.3 · p. 8
• The read-only, non-executing interface focuses evaluation on defensive source-code analysis rather than operational exploitation.Source: §3.3 · p. 9

4.1. Models Evaluated

• The evaluation covers 27 models and four static-analysis tools, spanning frontier, open-weight, and specialized systems.Open-weight models range from 350M to 753B parameters; systems use a fixed interface, and reported scores average three independent runs.Source: §4.1 · p. 9

4.2. Main Results

• Five highest-scoring model systems achieve at least 0.18 File F1, while all remaining systems score at most 0.16.Source: Table 2 · p. 10
• Performance varies across systems, and static-analysis tools outperform several language models despite limited coverage when relevant files are unknown.Static-analysis tools are displaced only by models fine-tuned for vulnerability localization; smaller models below 10B parameters do not match them without task-specific training.Source: §4.2 · p. 11
• Parameter count does not reliably order general-purpose models, and software-engineering capability is not a reliable proxy for cybersecurity performance.GPT-5.5 substantially outperforms GPT-5, while smaller GPT-5 Mini also exceeds GPT-5; GPT-5's system card reports greater conservatism on dual-use cybersecurity tasks, while GPT-5 Mini performs better on Cyber Range evaluations [36].Source: §4.2 · p. 11

4.3. False Positive Verification Results

• Phase B rankings differ from localization: strong localizers can report files in patched repositories, while high TNR can reflect abstention rather than clean-code recognition.False-positive control remains important for practical use because alert fatigue is a cybersecurity concern.Source: §4.3 · p. 11, Table 3 · p. 12

4.4. Scaling Analysis

• The size–File F1 relationship is positive but scattered and non-monotonic; VLoc Bench measures security-aware rather than general-purpose localization.The OLS trend covers 13 general-purpose open-weight models with available parameter counts. Local open-weight models up to 31B parameters complete evaluation in under one hour.Source: §4.4 · p. 13, Figure 2 · p. 13

5.1. Repository Difficulty Regression

• The repo-only regression explains R2=0.188 in-sample and R2=0.164 under cross-validation; concentrated code predicts easier localization, while larger or deeper repositories predict harder localization.Ground-truth file count carries negligible weight (−0.006), and severity and CWE category contribute negligible total weight.Source: §5.1 · pp. 14–15

5.2. Combined Regression: Structure Dominates

• Repository structure carries 4.5× more coefficient weight than model category; Elastic Net and permutation importance preserve the same qualitative ordering.The reported ratios are predictive associations, not causal contributions, and feature groups vary across different numbers of tasks and systems.Source: §5.2 · pp. 15–16
• Model descriptors explain R2=0.007 under cross-validation, while repository and model summaries leave 76% of observed variation unexplained.Model descriptors shift average performance but have limited ability to predict task-specific outcomes.Source: §5.2 · p. 16

5.3. Difficulty Decomposition

• Mean File F1 drops from 0.598 below 100 KB to 0.058 above 10 MB; Maven is hardest on average, pip and npm easiest, and Go lies between them.Go accounts for 43% of tasks, and ecosystem differences remain intertwined with repository size.Source: §5.3 · p. 17
• 38.4% of tasks receive File F1 = 0 from every model; large Go and Maven repositories account for 47% and 31% of this unsolved subset, respectively.Adding models or increasing parameter counts does not remove this subset under the evaluated settings.Source: §5.3 · p. 17

5.4. Behavioral Analysis

• Across models, higher File F1 is associated with more commands, targeted search and reading, and low tool-error rates; command volume alone is insufficient.The command-count relationship is correlational, and strong systems already use nearly their full command budget.Source: §5.4 · p. 18

5.5. Failure Mode Taxonomy

• The trace-based taxonomy gives each non-perfect trial one deterministic label; abstention accounts for 59.1% of failures, versus 40.9% for submitted failures.Exhausted budget (32.3%) and Wrong files (27.4%) are the two most common Level 2 categories.Source: §5.5 · p. 19
• Similar aggregate scores can conceal commit-heavy, cautious-exhausting, or premature-termination profiles, as well as partial recall on multi-file vulnerabilities.The taxonomy distinguishes these observable behaviors, which aggregate File F1 collapses.Source: §5.5 · pp. 19–20

6. Conclusion

• VLoc Bench evaluates both identifying files associated with a present weakness and recognizing a clean repository after the fix under the same task conditions.File F1 and TNR measure these complementary decisions.Source: §6 · p. 20
• The strongest system reaches 0.229 File F1, 38.4% of tasks get no correct localization, and strong localization does not ensure restraint on patched code.Systems that locate vulnerable files effectively can still report unsupported locations after remediation.Source: §6 · pp. 20–21
• Repository structure explains more performance variation than model descriptors, while parameter count alone is only a moderate guide.Source: §6 · p. 20
• Unlike benchmarks centered on exploitation or reproduction, VLoc Bench measures source-code localization from a CWE-only prompt and read-only repository access.Phase B tests whether agents avoid false positives after remediation.Source: §6 · p. 21

7. Limitations and Future Work

• Patch-touched implementation files provide a reproducible target but may omit relevant files, and a patched snapshot confirms remediation of the recorded vulnerability only.A finding outside the benchmark target cannot be adjudicated without an additional oracle.Source: §7 · p. 21
• Phase B conflates explicit abstention with turn exhaustion, API errors, or refusals when no submission is produced.Source: §7 · p. 21
• Future work will test richer vulnerability descriptions, validate additional Phase B findings with behavioral or other oracles, and evaluate intermediate security-analysis capabilities.Source: §7 · p. 21

8. Ethics and Responsible Use

• The benchmark evaluates localization rather than exploitation, uses only public advisories for already-patched vulnerabilities, and releases no exploits.Source: §8 · p. 21
• Automated localization outputs should be treated as leads for human review, not conclusions, and should augment rather than replace expert review.The included projects’ attack surface is not expanded beyond their public advisory record.Source: §8 · pp. 21–22

A. Example Task

• In the representative yahoo/serialize-javascript task, the agent receives a CWE-79 prompt and read-only repository access, with index.js as the patch-derived target.Source: A. Example Task · p. 26
• One run lists repository files, reads index.js in numbered segments, searches serialization and escaping logic, and submits [index.js] for File F1 = 1.0; Llama-3.3-70B exhausts its terminal budget without submitting, scoring 0.0.The runs differ in how they allocate the same interface and command budget; success depends on connecting the CWE description to relevant code and committing the correct file list.Source: A. Example Task · p. 26

B. Dataset Composition

• The 500-task sample is naturally composed, not balanced: Go is 43%, Maven 21%, npm 18%, Composer contributes one task, and High and Medium severity account for 83%.Source: B. Dataset Composition · p. 26

• The public release includes the benchmark and code needed to reproduce its evaluation, including the manifest, snapshot tools, harness, scoring implementation, and analysis scripts.The manifest records repository metadata, commit SHAs, CWE labels, ecosystem tags, and content-based MD5 checksums.Source: C. Data and Code Availability · pp. 26–27
• Go accounts for 43% of tasks, while Composer contributes one task; High- and Medium-severity tasks together account for 83%.Source: Figure 5 · p. 27, Figure 6 · p. 27
• All 500 tasks can be downloaded from public GitHub repositories at recorded commit SHAs without gated access or authentication.Source: C. Data and Code Availability · p. 27

D. Evaluation Cost and Runtime

• Anthropic models are omitted because evaluation exceeds the available budget; a 500-task sweep is estimated to cost over $600 for Claude Opus 4.8 or $1,658 with unconstrained Claude Code subagents.Future benchmark versions will incorporate these models as budget permits.Source: D. Evaluation Cost and Runtime · p. 28
• Open-weight models use vLLM on H100 GPUs, while closed-source models use provider APIs; the system prompt, tools, agent loop, and command budget are controlled across models.Provider defaults are not fully documented; static-analysis tools use default rulesets, except Semgrep-CWE uses task-matched CWE-specific rules.Source: D. Evaluation Cost and Runtime · p. 28
• Table 8 reports wall-clock runtime and estimated cost for a full 500-task Phase A evaluation, with local-model costs based on H100 rental rates and API costs based on June 2026 pricing.Source: Table 8 · p. 28

E. Lasso Regression Feature List

• The combined Lasso regression uses 67 standardized features: 52 repository-level features extracted per vulnerable snapshot and 15 model-level features constant for each system.Source: E. Lasso Regression Feature List · p. 28, Table 9 · p. 29
• Repository features vary across tasks while model features repeat across a system’s tasks, supporting repository-only, model-only, and combined specifications.Source: E. Lasso Regression Feature List · p. 28