PaperlayerView exact revision on arXiv

Source-linked AI summary

VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities

Jiahao Shi, Edward Tsien, Yifeng Di, Hongjiao Zhang, Yuan Tang, Ronit Dey, Ilona Shishov, Gal Netanel, Zvi Grinberg, Vladimir Belousov, Bat-Zion Rotman, Ilan Pinto, Tianyi Zhang

arXiv:2609.08040v1cs.CRcs.LGcs.SE

TL;DR

Coarse dependency matching raises false alerts, leaving analysts to assess exploitability case by case, and prior agent benchmarks do not evaluate known dependency vulnerabilities in downstream projects. VEX-BENCH introduces 75 expert-labeled real-world cases and evaluates nine models across three harnesses. Agents perform better on binary status classification than fine-grained justification; only GPT-5.5 exceeds 70% macro-F1 on justification.

  • Problem

    Coarse package-and-version matching generates false alerts and burdens analysts, while existing agent benchmarks do not assess known dependency vulnerabilities in downstream projects.

    Source: §1 · pp. 1–2

  • Method

    VEX-BENCH pairs real-world project codebases with known dependency vulnerabilities and evaluates agents’ structured status and justification predictions using expert-labeled cases.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Agents perform better on binary vulnerability-status classification than fine-grained justification, and only GPT-5.5 exceeds 70% macro-F1 on justification.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results indicate that identifying whether a project is affected is easier for these agents than providing an accurate justification.

    Source: §5 · p. 9

    The 75 cases cover only Go, Python, and Java and four not-affected justification types, so results may not capture the full diversity of practical fine-grained assessment.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing defenses such as GitHub Dependabot often raise many false alerts because their coarse-grained matching cannot determine whether a vulnerable dependency is actually exploitable. Security analysts typically spend substantial time assessing vulnerability exploitability case by case. Recent LLM agents have emerged as promising candidates for this task given their advanced capabilities in coding and cybersecurity, yet no existing benchmark evaluates them on it. Prior benchmarks target zero-day settings, where agents detect and exploit previously unknown vulnerabilities. In contrast, software supply chain security focuses on how known vulnerabilities in upstream dependencies affect downstream projects. This requires agents to reason across repositories and determine whether an upstream vulnerability is exploitable in the downstream project. To address this gap, we introduce VEX-Bench, the first benchmark for evaluating LLM agents' ability to assess the exploitability of software supply chain vulnerabilities. It contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go. We evaluate nine models across three agent harnesses. While GPT-5.5 and Claude Opus 4.6 reach approximately 80% F1 on binary vulnerability-status classification, only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification. This gap highlights the challenge of moving beyond binary exploitability assessment to identifying fine-grained exploitability reasons. Code and data: https://github.com/steven1518/vex-bench

1 Introduction

Software supply-chain vulnerabilities can propagate through complex dependencies, while coarse package-and-version matching produces false alerts that burden analysts. VEX-BENCH evaluates whether LLM agents can assess downstream exploitability, and results show that fine-grained justifications remain harder than binary status decisions.

1 Introduction

  • Complex package dependencies can propagate a vulnerable component’s risk to many downstream projects.The paper cites rapid growth in supply-chain attacks and Log4Shell’s broad impact as context for this exposure.Source: §1 · p. 1
  • Package-name and version matching can generate false alerts because it omits contextual information and fine-grained reachability analysis, adding to analysts’ triage burden.The paper notes that spurious alerts can bury critical vulnerabilities.Source: §1 · p. 1
  • Existing cybersecurity benchmarks primarily evaluate agents discovering or exploiting unknown vulnerabilities within a single codebase, unlike known dependency vulnerabilities affecting downstream projects.The latter setting requires combining external vulnerability evidence with cross-repository reasoning.Source: §1 · pp. 1–2
  • VEX-BENCH is the first benchmark for evaluating LLM agents’ assessment of software supply-chain vulnerability exploitability.It contains 75 expert-annotated real-world cases mined from GitHub across Go, Python, and Java.Source: §1 · p. 2
  • 81.6% status F1 for Claude Opus 4.6 and 79.9% for GPT-5.5 contrast with only GPT-5.5 exceeding 70% macro-F1 on fine-grained justifications.The authors report that precise exploitability reasons are more challenging than affected/not-affected classification.Source: §1 · p. 2

2 Related Work

Prior work addresses imprecise dependency alerts through reachability analysis, but reachability alone does not establish exploitability. Existing agent benchmarks focus on first-party vulnerabilities, while VEX-BENCH evaluates known dependency vulnerabilities in downstream projects.

2.1 Software Supply Chain Security

  • Reachability filtering can remove alerts for uninvoked vulnerable functions, but identifying those functions and building sound, precise call graphs are difficult, and exploitability may also depend on configuration and environment.VEX-BENCH evaluates end-to-end exploitability assessment of vulnerable dependencies in downstream projects.Source: §2.1 · p. 2

2.2 Cybersecurity Benchmarks for Agents

  • Existing cybersecurity benchmarks cover CTF challenges and real-world first-party vulnerabilities, asking agents to detect, exploit, or patch flaws in the codebase containing them.The cited real-world sources include historical CVEs, OSS-Fuzz reports, and bug-bounty disclosures.Source: §2.2 · p. 2
  • VEX-BENCH instead tests whether an agent can combine upstream vulnerability information with downstream dependency usage to decide whether a project is affected by a known CVE.The benchmark addresses dependency-level exploitability rather than vulnerabilities originating in the target codebase.Source: §2.2 · p. 2

3 VEX-BENCH

VEX-BENCH tests end-to-end assessment of known dependency vulnerabilities using real-world projects and expert-labeled cases. Its task combines external vulnerability research with code analysis, and its statistics show that most benchmark cases are not affected, with reasons varying across categories.

3 VEX-BENCH

  • Agents assess whether a downstream project is affected by a known dependency vulnerability, combining local code analysis with information retrieved from external sources.The assessment is communicated through the Vulnerability Exploitability eXchange (VEX) format.Source: §3 · pp. 2–3

3.1 Task Formulation

  • Each task supplies a complete target codebase and a CVE, but no curated advisory, preselected vulnerable files, or precomputed call-graph slice.Agents must retrieve vulnerability information and locate relevant code paths themselves, approximating the information available to a human expert triaging a flagged repository.Source: §3.1 · p. 3
  • Agents return a binary vulnerability status, a fine-grained reason for not-affected cases, and a reasoning trace recording inspected call sites, input paths, and mitigations.The trace makes the prediction auditable, while the labels distinguish reasons such as unreachable code and configuration preconditions.Source: §3.1 · pp. 3–4

3.2 Benchmark Construction

  • Construction mines CVE-referencing pull requests from popular repositories, then filters out test/example-only changes, cases with more than five CVEs, and cases lacking dependency-manifest changes.The manifest or lockfile modification is treated as evidence that the host project consumes and responds to the vulnerable dependency.Source: §3.2 · p. 4
  • Five experienced annotators calibrated labels on 15 cases, then labeled separate sets with peer review; the process produced 75 cases across Go, Python, and Java.Initial Fleiss’ Kappa was 0.667, and 88.3% of initial labels were confirmed without revision.Source: §3.2 · p. 4

3.3 Benchmark Statistics

  • The benchmark covers 75 cases, 67 CVEs, and 35 repositories; median project size is roughly 273K LOC, with cases ranging from 9,902 LOC to about 4.0M.Full projects rather than curated snippets require agents to locate relevant dependency call sites within large codebases.Source: Table 1 · p. 4, §3.3 · p. 4
  • 70.7% of cases are not affected and 29.3% are affected; code_not_reachable accounts for 38 cases, or 50.7% of the benchmark and 71.7% of not-affected cases.The other not-affected cases fall across configuration, missing-code, and environment reasons, motivating fine-grained labels.Source: Figure 2 · p. 5, §3.3 · p. 5

3.4 Defense Against Benchmark Hacking

  • Experiments use isolated containers separated from labels and evaluation artifacts, standalone snapshots without Git history, and no executable test cases.The authors also report that manually annotated ground truth is not publicly available and found no meaningful evidence that source pull requests expose benchmark answers or that evaluated models memorized them.Source: §3.4 · p. 5

4 Evaluation

The evaluation finds a consistent gap between binary vulnerability-status prediction and fine-grained justification, comparing agents across traditional tools, cost, languages, error patterns, repository sizes, and harnesses.

4.1 Experimental Setup

  • Nine models were evaluated under vendor-specific or OpenCode harnesses, using binary status and five-class justification metrics across three runs.Experiments used isolated language-specific Docker environments and a 10-minute task limit; 98.2% of executions finished within the limit.Source: §4.1 · pp. 5–6

4.2 Main Results

  • GPT-5.5 led all three justification metrics, with 73.5% macro-F1; Claude Opus 4.6 led status F1 at 81.6%, while justification accuracy was lower than status accuracy for all nine configurations.GPT-5.4 mini had the largest accuracy drop: from 82.2% status accuracy to 66.7% justification accuracy, a gap of 15.5 percentage points.Source: §4.2 · p. 6

4.3 Comparison with Traditional SCA Tools

  • Reachability analysis did not close traditional scanners’ false-positive gap: 70% of reachability tools’ false positives were labeled code_not_reachable.A call-graph appearance does not establish feasible paths or triggerable branch conditions; govulncheck completed only 19 of 30 Go cases because projects had to build successfully.Source: §4.3 · p. 7, Table 3 · p. 7

4.4.1 Cost-Performance Analysis

  • 81.2% balanced performance made GPT-5.5 the strongest but most expensive configuration at $1.082 per case; DeepSeek-V4-Pro reached 74.4% for $0.075.Figure 3 compares per-case cost on a log scale and connects Pareto-efficient configurations; GLM-5.1 reached 74.9% at $0.095 per case.Source: §4.4.1 · p. 7, Figure 3 · p. 7

4.4.2 Language Analysis

  • Justification macro-F1 ranged from 56.5% on Python to 75.5% on Go, a 19.0-point spread, versus a 10.0-point status-F1 spread.Python was weakest on both measures; the authors suggest differences in language ecosystems may help explain the pattern.Source: §4.4.2 · pp. 7–8, Figure 5 · p. 8

4.4.3 Error Analysis

  • Errors concentrated in code_not_reachable and vulnerable cases, but models differed: GPT-5.5 often predicted requires_configuration for code_not_reachable cases.Four of five manually reviewed cases supported both explanations under different configurations; Claude Opus 4.6 often over-called such cases as vulnerable, while DeepSeek-V4-Pro’s boundary was less decisive.Source: §4.4.3 · p. 8, Figure 4 · p. 8

4.4.4 Repository Size Analysis

  • GPT-5.5 and Claude Opus 4.6 maintained status F1 of 85.0% and 84.5% in the largest repository windows, while justification scores were lower and more variable.DeepSeek-V4-Pro’s status F1 dropped from 83.2% in the smallest window to 71.0% in the largest; windows ranged from about 44K to 1.9M LOC.Source: §4.4.4 · pp. 8–9, Figure 6 · p. 9

4.4.5 Harness Ablation Study

  • Codex CLI outperformed OpenCode across GPT-5.5 metrics while using more than twice the tokens; for GPT-5.4 mini, performance was comparable but token use was roughly 4× higher.Figure 7 reports status and justification accuracy/F1 alongside average per-case token usage.Source: §4.4.5 · p. 9, Figure 7 · p. 9

5 Conclusion

VEX-BENCH evaluates whether LLM agents can assess exploitability of software supply chain vulnerabilities. Across nine model–harness configurations, agents perform better on binary status than on fine-grained justification.

5 Conclusion

  • Agents perform better on binary vulnerability-status classification than on fine-grained justification across nine model–harness configurations.The results suggest they can identify whether a project is affected but lack the depth needed for accurate justifications.Source: §5 · p. 9
  • VEX-BENCH comprises 75 real-world cases from Go, Python, and Java projects, with ground-truth labels manually annotated by security experts.The benchmark was designed to evaluate LLM agents’ assessment of software supply chain vulnerability exploitability.Source: §5 · p. 9
  • The paper concludes that agents’ binary affected-status judgments are more capable than their precise exploitability explanations.The conclusion characterizes accurate justification as requiring greater analytical depth.Source: §5 · p. 9

Limitations

The benchmark’s limited size and category coverage, single-label formulation, and reliance on repository snapshots constrain how fully its results represent real-world exploitability assessment.

Limitations

  • Expert annotation costs limit VEX-BENCH to 75 cases across three ecosystems, with sparse fine-grained categories, four not-affected reasons, and no JavaScript/npm coverage.The authors caution that results may not capture the diversity of practical challenges, especially for fine-grained justification.Source: Limitations · p. 9
  • A precedence rule assigns one gold label when a case admits multiple valid explanations, potentially omitting secondary reasons a project is not affected.This single-label formulation makes evaluation unambiguous but narrows what the label records.Source: Limitations · p. 9
  • Repository snapshots provide static source and configuration evidence, so judgments approximate documented or default conditions rather than every real deployment.They cannot fully capture dynamic user inputs or deployment-specific environment settings.Source: Limitations · p. 9

Ethical Considerations

The paper frames VEX-BENCH as defensive triage research while noting its dual-use risk. Its four benchmark justifications distinguish different reasons a downstream project may not be affected.

Ethical Considerations

  • VEX-BENCH could help malicious actors prioritize known vulnerabilities, but uses public CVEs, requires no exploit generation, and analyzes source snapshots in isolated Docker environments.The authors intend the benchmark for defensive triage to prioritize genuine risks and reduce false-positive alerts.Source: Ethical Considerations · p. 10

A Justification Category Definitions

  • The four not-affected labels distinguish absent vulnerable dependencies, unreachable vulnerable functions, disabled configuration paths, and unmet runtime-environment prerequisites.The categories require checking dependency presence, call paths, configuration defaults, or the project’s declared deployment environment, respectively.Source: A Justification Category Definitions · p. 12
  • Additional VEX-aligned reasons and boundary cases inform prompt design, but are not separate benchmark classes in reported statistics.The paper reports dataset statistics and justification metrics over the four benchmark categories.Source: A Justification Category Definitions · p. 12, Table 5 · p. 13
  • The benchmark uses four not-affected justification categories adapted from CISA’s VEX vocabulary.Table 4 summarizes these benchmark categories.Source: A Justification Category Definitions · p. 12, Table 4 · p. 13

B Prompts

The appendix prompt guides agents through static, evidence-based classification of a known vulnerability in a repository. It specifies category precedence, output expectations, and how prompt categories map to the evaluated labels.

B.1 Task Instruction Prompt

  • The prompt treats the working directory as a source tree and directs agents to use visible evidence rather than assume unsupported runtime behavior.The task context excludes a running process and deployment context.Source: §B.1 · p. 13
  • Agents must choose the first applicable category in a fixed precedence order rather than skip to a later explanation.For example, absence takes precedence over reachability or environment factors.Source: §B.1 · pp. 13–14
  • The prompt’s reasons include package-mapping errors, absent or unreachable code, missing prerequisites, protective controls, uncertainty, and vulnerability.The broader prompt category space contains 12 fixed categories.Source: §B.1 · pp. 13–14
  • A case is vulnerable only when affected code is present, used, reachable from an attack surface, and not effectively mitigated.The prompt specifies these as jointly required conditions.Source: §B.1 · p. 14
  • If vulnerability criteria fail, the prompt selects the first matching non-vulnerable category; uncertainty is reserved for genuinely insufficient evidence.The category order governs which non-vulnerable explanation is selected.Source: §B.1 · p. 14
  • The required output is a single JSON object with a category and evidence-based reasoning tied to concrete repository or advisory details.The example grounds its explanation in an affected version and first-party source inspection.Source: §B.1 · p. 14
  • Evaluation uses five annotated classes: vulnerable and four not-affected reasons; predictions in other prompt categories count as FAILED.Vulnerable maps to affected status, while the four justification categories map to not-affected status.Source: §B.1 · p. 14

C Model-Side Leakage Analysis

The authors assess whether public source pull requests could reveal VEX-BENCH answers through direct disclosure or model memorization. Their audit and completion probe provide no meaningful evidence that the evaluated models could recover those answers.

C Model-Side Leakage Analysis

  • Only three of the 75 source PRs discuss exploitability, and none reveals the annotators’ vulnerability status or justification.The audit reviewed PR titles, descriptions, and discussion threads.Source: C Model-Side Leakage Analysis · p. 14
  • The guided-completion probe selected 19 human-written, pre-cutoff PRs and tested GPT-5.5 and Claude Opus 4.6 without web access or tools.PRs were selected for being non-boilerplate and long enough for a meaningful completion test.Source: C Model-Side Leakage Analysis · pp. 14–15
  • Across 38 completions, there were no verified near-exact matches and one exact ten-token continuation that repeated the prompt’s PR title.Together with the source-PR audit, the results provide no meaningful evidence of memorized PR content or recoverable benchmark answers.Source: C Model-Side Leakage Analysis · p. 15

D Experimental Details

The experimental setup specifies container environments, model backends, and harness configurations. The paper reports these implementation choices separately to distinguish model identifiers from agent-side execution settings.

D.1 Environment

  • The Docker runtime table reports each language environment’s base image, operating system, and primary runtime or build-tool versions.Common utilities are inherited or installed during container setup.Source: §D.1 · p. 15

D.2 Model Backends

  • All model API calls used Azure’s model-serving APIs through a common REST interface, with model names, release sources, and public release dates listed separately.The model backends are reported separately from tool versions.Source: §D.2 · p. 15, §D.3 · p. 15

D.3 Harness Framework

  • The harness table records framework documentation, installed CLI versions, and effort settings; Codex used xhigh, Claude Code high, and OpenCode no override.High was the maximum supported effort for the Claude 4.6 models; OpenCode had no analogous configured setting.Source: §D.3 · p. 15

D.4 Runtime and Timeout Analysis

Most agent executions finish within the runtime cap, though runtime varies across configurations and reflects the full execution stack. The benchmark is intended for defensive triage research, not exploitation.

D.4 Runtime and Timeout Analysis

  • 98.2% of executions finish within the 10-minute limit, while timeouts cluster in Kimi K2.6 and MiniMax-M2.7 for different observed reasons.Kimi timeouts mainly involve long analyses on large repositories; MiniMax timeouts are primarily associated with slow API responses. Runtime combines backend, harness, and tool-trajectory latency.Source: §D.4 · p. 15

E.1 Artifact Use and Intended Use

  • VEX-BENCH is intended for defensive triage, agent-reliability evaluation, and failure-mode analysis, not exploit generation or attacks on deployed systems.Users remain responsible for complying with licenses and terms of upstream repositories, tools, models, and vulnerability-data sources.Source: §E.1 · p. 15

E.2 Data Privacy and Offensive Content

The paper describes privacy protections, dual-use risks and safeguards, annotation procedures, and the limited role of AI assistants. These practices frame the benchmark as a research resource built from public artifacts and expert judgments.

E.2 Data Privacy and Offensive Content

  • The benchmark uses public repositories and vulnerability records, excludes person-centered materials, removes Git history, and reviews metadata for unnecessary personal or unrelated sensitive content.Released metadata is intended to contain task and repository references, CVE identifiers, labels, annotations, prompts, and evaluation scripts rather than contributor or annotator personal information.Source: §E.2 · pp. 15–16

E.3 Potential Risks

  • The benchmark has dual-use potential because exploitability assessments could help prioritize known vulnerabilities, but tasks use disclosed issues and require neither payloads nor end-to-end attacks.Static source snapshots, isolated containers without deployment context, and prompts directing agents to code and advisory evidence are described as risk-reduction measures.Source: §E.3 · p. 16

E.4 Annotation Instructions and Consent

  • Annotators assessed dependency presence, vulnerable-function use and reachability, and in-repository mitigations; affected labels required all conditions to hold.Otherwise, they selected the most specific not-affected justification among four reported categories: missing vulnerable code, unreachable code, required configuration, or required environment.Source: §E.4 · p. 16
  • Labels had to rest on concrete code and advisory evidence; ambiguous cases were discussed during calibration and disagreements reconciled before final assignment.Annotators were research-team members informed of the use of their annotations, while identities and annotator-level performance were not released.Source: §E.4 · p. 17

E.5 Use of AI Assistants

  • AI assistants supported language polishing, manuscript revision, and literature searches, but not ground-truth labeling, final exploitability judgments, or experimental results.Source: §E.5 · p. 17
Loading 2609.08040v1…