PaperlayerView exact revision on arXiv

Source-linked AI summary

A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks

Maciej Cichoń, Bartłomiej Dmitruk

arXiv:2609.32890v2cs.LG

TL;DR

Published vulnerability-detection evaluations report differing scores for similar models, and the contribution of evaluation protocol to that variation is unclear. The study holds model outputs fixed while varying metric, verdict extraction, and output budget, and evaluates models across released pair benchmarks and a pooled set. For 37 of 68 models the net score is within its interval of the null’s zero, while both-flagged and both-cleared rates exceed the null’s.

  • Problem

    Published vulnerability-detection evaluations report differing scores for similar models, and the contribution of evaluation protocol to that variation is unclear.

    Source: §1 · p. 1

  • Method

    The study holds model outputs fixed while varying metric, verdict extraction, and output budget, and evaluates models across released pair benchmarks and a pooled set.

    Source: §1 · pp. 1–2

  • Results

    For 37 of 68 models the net score is within its interval of the null’s zero, while both-flagged and both-cleared rates exceed the null’s.

    Source: §6 · p. 8

  • Takeaways & Limitations

    A function-level or commit-level score measures flag rate more than model, so the paper recommends reporting pair-correct with outcome rates, net score, null, and interval.

    Source: §6 · p. 8

    The activation probe is supervised, and whether models use its direction was not tested.

    Source: §7 · p. 8, §5 · p. 8

Abstract

from arXiv · show

Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured how much of that difference evaluation protocol accounts for, with model outputs held fixed. In a paired test, a model must flag a vulnerable function and clear its version after a fixing commit. Three choices that published evaluations make differently were varied one at a time: metric, verdict extraction and output budget. Seven frontier and large open models were evaluated on five released pair benchmarks and a set pooled for this work under one protocol, and 61 open models of 1.5B to 36B parameters on the pooled set. Function-level F1 follows how often a model flags both functions of a pair (Spearman $+0.86$ over 42 combinations) and is nearly unrelated to pair-level correctness ($+0.16$). On the pair score, extraction changes a model's number by $+0.001$ at the median and budget by $+0.02$ with an interval through zero, whereas the model changes a benchmark's number by up to 0.18 and the benchmark a model's by up to 0.16; a function-level score therefore measures flag rate more than model. For 37 of 68 models the difference between correct and reversed pairs is within its 95% interval of zero, the value for a null model that flags each function at a fixed rate, while both-flagged and both-cleared rates exceed that null by 0.055 on median, and for 64 of 68 both functions of a pair receive one answer more often than independence predicts: verdicts are determined by the text common to both functions. On length-matched pairs a linear probe on activations separates 0.78 by within-pair ranking, against 0.64 for a tf-idf baseline and 0.5 for length; the generated verdict is near chance for three of six models and at 0.55 to 0.57 for the other three, and a prompted logit is at chance for all six.

1 Introduction

The paper asks how much vulnerability-detection scores depend on evaluation protocol when model outputs are fixed. Its findings show that function-level scores track flag rate more than model, while pair-level scores depend more on model and benchmark than on extraction or budget.

1 Introduction

  • In a paired test, a model is credited only when it flags the vulnerable function and clears its post-fix version.Earlier evaluations commonly reported one function-level accuracy or F1 number per model.Source: §1 · p. 1
  • The study varies metric, verdict extraction, and output budget one at a time while holding model outputs fixed.Metrics include function-level accuracy or F1 and pair outcomes; extraction uses either a required verdict line or a language-model judge, and budgets differ across evaluations.Source: §1 · pp. 1–2
  • Seven API models were evaluated on five released pair benchmarks and a pooled set, while 61 open models spanning 1.5B to 36B parameters were evaluated on the pooled set.For six open models, the same pairs were also scored with a prompted logit, activation oracle, and supervised linear probe.Source: §1 · p. 2
  • Function-level scores measure flag rate more than model, while pair-level scores depend more on model and benchmark than on extraction or budget.The paper also reports that verdicts are determined by text common to both functions.Source: §1 · p. 2

2 Related work

Prior work documents problems in security-fix datasets and introduces paired benchmarks, while related studies examine protocol effects and information decoded from model activations.

2 Related work

  • Security-fix datasets have documented label errors, duplication, exploitable regularities, and vulnerabilities dependent on calling context.Croft et al. [2023] found 20 to 71 percent of labels inaccurate; Chakraborty et al. [2022] and Steenhoek et al. [2023] showed classifiers exploit dataset regularities; Risse et al. [2025] found many inspected functions vulnerable only through calling context.Source: §2 · p. 2
  • PrimeVul [Ding et al., 2025] introduced paired testing, followed by VulnPatchPairs [Risse and Böhme, 2024], PairVul [Du et al., 2024], JitVul [Yildiz et al., 2025], and SecLLMHolmes [Ullah et al., 2024].Earlier pair-level results placed frontier models within four to six points of 8B and 32B open models on JitVul and PairVul; on CWE-Trace, models from 4B to GPT-4.1-mini scored 49 to 54 percent (Appendix G).Source: §2 · p. 2
  • Prior work shows protocol choices can change evaluation effects [Schaeffer et al., 2023], while extraction studies report regex misses and near-chance language-model judging on hard items [Yu et al., 2025; Tan et al., 2025].Linear probes reveal information decodable from activations, which is not necessarily information the model uses [Alain and Bengio, 2016; Hewitt and Liang, 2019; Belinkov, 2022; Ravichander et al., 2021].Source: §2 · p. 2

3 Setup

The setup evaluates paired vulnerability benchmarks under a shared protocol and defines pair-level outcomes alongside function-level metrics and a fixed-rate null. It includes API and open models, with a pooled dataset assembled from several sources.

3 Setup

  • The pooled benchmark contains 5,705 C/C++ pairs from PrimeVul, ARVO vulnerabilities with located fixes, and author-extracted CVE-fixing commits.Its 1,541-pair execution-verified tier combines ARVO pairs with 133 reproduced commit pairs; the other 4,164 pairs have commit-heuristic labels.Source: §3 · p. 2
  • Models received the same prompt, sampled each pair side three times at temperature 0.7, and used majority verdicts; ties and missing verdict lines were recorded separately.The output budget was 32,768 tokens, and benchmark draws were capped at 200 pairs except SecLLMHolmes, for which all 33 were used.Source: §3 · pp. 2–3
  • Table 1 compares the effects of protocol choices on pair-correct with the effects of model and benchmark; its intervals are 95% bootstrap intervals over pairs.Source: Table 1 · p. 3
  • Each pair yields four outcomes: P-C, vulnerable side flagged and patched side cleared; P-V, both flagged; P-B, both cleared; and P-R, reversed.Pair-correct is the P-C rate, while the net score is P-C minus P-R; function-level accuracy and F1 treat the 2n sides as independent items.Source: §3 · p. 3
  • The reference null flags each side independently with probability p, giving equal P-C and P-R rates and a net score of zero.At p = 0.5, its pair-correct is 0.25; on balanced sides its F1 is 0.67 when it flags everything.Source: §3 · p. 3
  • The study included seven API models and 61 self-hosted open models from 1.5B to 36B parameters, with six open models also having released activation oracles.The 61 open models span thirteen base families and were evaluated on the 1,000-pair draw.Source: §3 · p. 3

4 Results

Across identical model outputs, function-level F1 tracks how often both sides are flagged more closely than whether pairs are correctly classified. Extraction and output-budget changes are small or uncertain overall, while model and benchmark differences remain substantial.

4.1 The metric

  • Across 42 combinations, Spearman correlation is +0.86 between F1 and both-flagged rate (bootstrap interval +0.74 to +0.92), versus +0.16 between F1 and pair-correct (interval −0.16 to +0.46).F1 ranges from 0.36 to 0.68, while pair-correct ranges from 0.06 to 0.25. For 37 of 68 models, the observed net-score interval includes zero; both-flagged and both-cleared rates exceed their independent-null values by 0.055 on median.Source: §4.1 · pp. 3–4

4.2 Verdict extraction

  • Extraction changes pair-correct by +0.001 at the median, but judge disagreements and missing verdict lines can shift scores for particular models.On samples with verdict lines, three judges agree with them at median rates of 96.8%–99.9%; parsing failures in three formats lead to shifts of 0.03–0.10 under a judge or tolerant parse.Source: §4.2 · p. 5

4.3 Output budget

  • At 1,024 versus 32,768 tokens, gpt-5.2’s pair-correct is 0.067 versus 0.089, a +0.022 difference with an interval spanning zero.The budget needed to keep missing verdict lines below 1% varies by model: 4,096 tokens for gpt-5.2, 32,768 for deepseek-r1, and above 32,768 for DeepSeek-V4-Flash.Source: §4.3 · p. 5

4.4 Benchmarks and the open population

  • The six benchmarks agree on the coarse model order (Kendall’s W = 0.65), but distinguish only 3 of 36 adjacent model pairs.The pooled set agrees with PrimeVul at 0.93, partly by construction; among open models, 15 exceed a net score of 0.05, while 37 of 61 have intervals including zero.Source: §4.4 · pp. 5–6, Figure 2 · p. 6

5 Four methods of obtaining a verdict from the same model

The section compares four ways to score the same model outputs, finding that activation probes rank vulnerable functions above their patches far more often than generated verdicts or prompted logits. This ranking signal persists on length-matched pairs, but whether the model uses it was not tested.

5 Four methods of obtaining a verdict from the same model

  • 0.81–0.82 probe ranking on six models exceeds generated verdicts at 0.50–0.57 and prompted logits at 0.45–0.55.The comparison uses within-pair ranking, with the vulnerable side scored above its patch.Source: Figure 3 · p. 7, §5 · p. 7
  • The comparison scores a prompted yes/no logit, an activation oracle, and a supervised linear probe by within-pair ranking, with tf-idf as a baseline.Ranking requires a direction that moves with the patch but no threshold; absolute accuracy is not reported.Source: §5 · pp. 6–7
  • On 564 pairs with sides differing in length by under 5%, probe ranking has a median of 0.78, against 0.64 for tf-idf.On 125 pairs where the patch removed code, probe ranking falls to a median of 0.55.Source: §5 · p. 7
  • Whether the model uses the probe direction was not tested; the authors state that testing this would require an intervention along that direction.The probe signal follows code added by a patch, and probe accuracy without intervention is descriptive.Source: §5 · pp. 7–8

6 Discussion

The discussion argues that function-level and commit-level scores mostly measure flag rate, while pair-level correctness remains low and uncertain. It recommends reporting pair outcomes alongside the null score and uncertainty, and identifies open questions about shared-text dependence and the probe direction.

6 Discussion

  • For 37 of 68 models, net score falls within the null’s zero interval, while both-flagged and both-cleared rates exceed the null; 64 of 68 show same-answer dependence across pair sides.The discussion attributes verdicts to text common to both functions.Source: §6 · p. 8
  • The input-dependent net score lies between 0.03 and 0.22 for every model that has one, and the best 27B open model equals the best API model.The discussion says the flag rate removes most of the difference between models of very different size.Source: §6 · p. 8
  • A linear probe ranks 0.78 of length-matched pairs, while generated verdicts rank at 0.50–0.57; its signal follows code added by patches, a corpus regularity as well as a model property.The discussion notes that some pairs cannot be decided from the function alone and that human accuracy was not measured.Source: §6 · p. 8
  • The authors recommend reporting pair-correct with all four outcome rates, the null score, and a bootstrap interval, while reserving model rankings for paired intervals excluding zero.They also leave open whether intervention along the probe direction changes verdicts and what shared-text dependence corresponds to in the weights.Source: §6 · p. 8

7 Limitations

The limitations include uncertainty in pair-correct estimates, a supervised probe whose result is scoped to patches adding code, and evaluation restricted to C/C++ functions and one prompt. The prompt wording, temperature, and sampling choices were not varied, and contamination cannot be excluded for several public benchmarks.

7 Limitations

  • At n=200, pair-correct sampling noise is 0.009–0.022; at n=1,000, it is 0.004–0.011, making 0.03 model differences on a 200-pair benchmark noise-level.The estimate uses one sample per side and bounds noise for the majority of three from above.Source: §7 · p. 8
  • The probe is supervised, and on 125 pairs whose patch removes code it is at chance; all evaluated pairs are C/C++ functions.One prompt was used, prompt-wording variance was not measured, and temperature 0.7 and K=3 were held fixed.Source: §7 · p. 8
  • Four of five released benchmarks and ARVO are public and predate every evaluated model, so contamination cannot be excluded.Source: §7 · p. 8

A Prompt

  • The prompt requires an exact first-line vulnerable or not-vulnerable verdict followed by a short justification, and parsing takes the first verdict line after visible reasoning.Source: A Prompt · p. 11

B Benchmarks

Table 4 distinguishes the benchmark’s deduplicated pair count from the number evaluated per model, including different evaluation counts for the pooled set.

B Benchmarks

  • “Pairs” counts benchmark pairs after exact-duplicate removal.Source: Table 4 · p. 11
  • “Evaluated” counts pairs scored per model.Source: Table 4 · p. 11
  • For the pooled set, the panel evaluates 1,000 pairs, while the two full-corpus runs evaluate all 5,705.Source: Table 4 · p. 11

C The input-independent null per model

Across models, pair-correct minus reversed often cannot be distinguished from zero, while function-level F1 tracks both functions being flagged much more closely than pair correctness. Model and benchmark rankings vary, and the open-model population includes substantial variation in parsing and output budgets.

C The input-independent null per model

  • For 37 of 68 models, the 95% interval of pair-correct minus reversed includes zero; both-flagged and both-cleared rates are not below their model-specific null values.The null flags each side at the model’s own rate; Figure 4 shows most models within noise of it across flag rates.Source: C The input-independent null per model · p. 11, Figure 4 · p. 12

D Pair-correct minus reversed across the panel

  • 0.22 is the largest net score, for grok-4.3 on the pooled set; deepseek-v3.2 reaches −0.03 on PrimeVul.Across benchmarks, reversed rates range from 0.00 to 0.12, and qwen3-8b has the highest or joint-highest reversed rate on each benchmark.Source: D Pair-correct minus reversed across… · p. 11

E Function-level F1 across the panel

F The open-model population

  • Among 61 open models, output budgets range from 4,096 to 32,768 tokens, and four models have unparsed-sample rates above 5%.VulnLLM-R-7B has a 27% unparsed rate, the R1-0528 distill 19%, Ministral-3-8B 11%, and Ministral-3-3B 5%.Source: F The open-model population · p. 12

G Pair-level results in the literature

Prior pair-level evaluations report divergent model scores, while published function-level metrics can rank models differently from pair accuracy. The paper also examines how abliterated variants and judge-based verdict extraction behave under the evaluation protocol.

G Pair-level results in the literature

H The abliterated variants

  • Abliterated variants generally retain base-model discrimination within 0.03 in seven of eight cases, but their default answer shifts by model family.The gpt-oss-20b variant changes to flagging everything and loses its discrimination, from 0.13 to 0.00.Source: H The abliterated variants · p. 13

I Judge protocol

  • Judge extraction agrees with parsed verdicts on 99.2% to 100% of API-model samples that contain a verdict line, but may supply answers when no line exists.For example, grok-4.3 judged all 86 empty outputs “not vulnerable”; judge extraction changed pair-correct by −0.016 to +0.034 per model.Source: I Judge protocol · p. 13

J Length control for the probe

Length-matched pairs test whether probe performance reflects code content rather than function length. The probe outperforms tf-idf on that subset, but the pattern changes when patches remove code.

J Length control for the probe

  • 0.78 is the median probe score on 564 pairs whose sides differ in length by under 5%, versus 0.64 for tf-idf.Across those pairs, probe scores range from 0.72 to 0.83; mean-pooled activations depend on length, motivating the matched comparison.Source: J Length control for the probe · p. 13
  • When the vulnerable side is longer because the patch removed code, probe scores fall to a 0.55 median and tf-idf reaches 0.45.The authors conclude that the probe depends on code added by a patch; within the tested scope, its signal is not explained by length.Source: J Length control for the probe · p. 14

K Protocol for the four verdict methods

The protocol compares generated verdicts with three alternatives: a prompted logit, an activation oracle, and a supervised linear probe. It defines pair-level outcomes and reports the data and scoring conventions used for the four methods.

K Protocol for the four verdict methods

  • The linear probe uses standardized mean-pooled hidden states from every layer and logistic regression, evaluated by within-pair ranking with project-grouped folds.The probe uses five folds, out-of-fold scores, and 2,000 bootstrap resamples over pairs; tf-idf and function length are baselines on the same folds.Source: K Protocol for the four verdict methods · p. 14
  • The activation oracle scores the last 50 tokens at a model adapter’s middle layer, while the prompted logit scores Yes versus No after the function text.The oracle’s activations are truncated to include the patched region and injected at layer 1; the adapter-on-text control omits the injection hook.Source: K Protocol for the four verdict methods · p. 14
  • Pair outcomes are P-C, P-V, P-B, or P-R; pair-correct is P-C, and the net score is P-C minus P-R.Function-level accuracy and F1 treat the 2n sides as independent items; Table 5 compares observed rates with a null model using each model’s flag rate.Source: Table 5 · p. 15
  • The protocol’s visual summaries include four model outcomes sorted by net score, population outcome rates and F1, and model budgets and missing-verdict rates.The open-judge table reports agreement with parsing and pair-correct differences; the four-method table compares verdict net score and within-pair rankings, with chance at 0.5 and tf-idf at 0.728.Source: Figure 5 · p. 17, Table 8 · p. 18, Table 9 · p. 19

Related papers

Loading 2609.32890v2…