PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, Yazan Boshmaf

arXiv:2609.08765v1cs.CRcs.AIcs.CL

TL;DR

Cybersecurity LLM benchmark scores are often treated as stable despite depending on configurable evaluation pipelines, creating a reliability gap in a high-stakes domain. The paper models benchmarks as measurement pipelines and audits eight benchmarks across 10 LLMs, finding systematic failure modes, large score shifts, and unstable rankings. It concludes that pipeline-aware auditing and explicit evaluation choices are needed for reliable benchmarking.

  • Problem

    Benchmark scores are often interpreted as stable measurements of capability even though configurable prompts, inference, extraction, scoring, and aggregation can affect them.

  • Method

    The paper formalizes benchmarks as measurement pipelines and audits eight cybersecurity benchmarks across 10 LLMs using controlled pipeline perturbations and a standardized evaluation harness.

  • Results

    Across 23 tasks, the audit identifies 15 recurring failure modes; a pipeline choice can shift scores by more than 80 percentage points, while similar tasks can rank models differently.

  • Takeaways & Limitations

    Cybersecurity benchmark scores should be treated as pipeline-dependent measurements supported by meaningful capability coverage and explicit, consistent evaluation pipelines.

  • Takeaways & Limitations

    Comparative conclusions remain conditional on defensible but non-unique choices for extraction, invalid-response denominators, partial credit, and scoring method.

Abstract

from arXiv · show

Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.

1 Introduction

Cybersecurity LLM benchmark scores are not fixed measurements: configurable pipeline stages can alter scores, rankings, and deployment-relevant conclusions. The paper audits these effects across benchmarks and models, then motivates standardized, reproducible evaluation.

  • Motivation: Benchmark outcomes may reflect the measurement process as much as underlying model capability.Evaluation scripts configure prompts, inference, extraction, scoring, and aggregation, so scores are conditional on these choices.
  • Audit scope: The audit covers eight cybersecurity benchmarks, 48,662 questions, 23 tasks, and 10 LLMs, identifying 15 recurring pipeline failure modes.The models include proprietary, open-weight, and cybersecurity-specialized systems.
  • Pipeline effects: 85.9 percentage points are recovered for Qwen3.6 by correcting a newline stop sequence, allowing reasoning to finish and stripping the reasoning span before extraction.The original configuration halted generation inside the reasoning preamble, producing empty outputs.
  • Cross-benchmark reliability: Kendall’s τ-b rank agreement is only 0.29 and 0.24 for two semantically similar task pairs because scoring conventions differ.The cited discrepancies involve binary versus partial-credit scoring and alias handling.
  • Standardization: Nine of 10 models shift by at least three ranks on at least one benchmark under a harness that standardizes pipeline fields while preserving task semantics.The harness is diagnostic rather than a universally correct evaluator, and observed failure rates are specific to the studied cybersecurity benchmarks.
  • Implications: The paper recommends treating benchmark scores as pipeline-dependent measurements and adopting audits, executable reference evaluators, invalid-response reporting, and reliability statistics.These practices are presented as ways to make evaluation choices explicit and reproducible.

2 Benchmarks as Measurement Pipelines

The paper models each benchmark as a nested measurement pipeline whose stages transform data and model outputs into scores. It audits stage-specific choices through controlled perturbations and evaluates both within-benchmark effects and cross-benchmark ranking stability.

  • Pipeline model: A benchmark score is the output of a composed pipeline rather than an intrinsic property of the model alone.The pipeline includes dataset construction, prompt specification, inference, extraction and scoring, and aggregation.
  • Pipeline effects: Variation at any pipeline stage can change reported performance without reflecting a change in model capability.Extraction converts outputs into predictions or graded judgments, while aggregation combines question- and task-level measurements.
  • Pipeline stages: Nine configuration fields instantiate the five stages, covering prompts, decoding, token limits, stop sequences, extraction, scoring, denominators, and aggregation.These fields are specified, perturbed, and standardized throughout the paper.
  • Audit procedure: The audit reconstructs documented and released pipelines, records disagreements and unspecified choices, and logs raw outputs with prompts and evaluation settings.This enables controlled comparisons while preserving the exact conditions used to produce scores.
  • Perturbation analyses: Re-scoring changes downstream fields while holding model outputs fixed, whereas re-generation changes upstream fields and produces new outputs.Re-scoring therefore isolates the modified downstream field more directly than re-generation.
  • Cross-benchmark analysis: PCA measures redundancy in task-level scores, while Kendall’s τ-b and pairwise inversions measure agreement among task-induced model rankings.These analyses distinguish instability within benchmarks from inconsistent comparative conclusions across tasks.

3 Experimental Setup

The experimental setup evaluates cybersecurity benchmarks spanning diverse tasks and authoritative data sources against 10 heterogeneous LLMs. A common harness standardizes pipeline fields where semantics permit and records evaluation traces for reproducibility.

  • Study design: The study evaluates eight cybersecurity benchmarks and 10 LLMs, measuring how pipeline choices affect scores and comparative conclusions.The models span proprietary, open-weight, and cybersecurity-specialized categories.
  • Benchmark coverage: The benchmarks cover multiple-choice knowledge, vulnerability scoring, root-cause mapping, threat-actor attribution, attack-technique extraction, mitigation selection, and security reasoning.Their sources include CVE, CWE, CVSS, MITRE ATT&CK, advisories, and threat-intelligence reports.
  • Data selection: The audit scores each benchmark using its full released dataset except when release structure or validation protocols require a subset.For SecBench, the study scores 661 English MCQs from a reported 47,910 questions.
  • Evaluation harness: The harness standardizes nine pipeline fields wherever benchmark semantics permit without changing questions, gold answers, or intended capabilities.This separates evaluation-procedure corrections from changes to the underlying task.
  • Reproducibility: The harness records raw outputs, extracted predictions, question-level and aggregate scores, invalid-response rates, and pipeline configuration.These traces make differences between original and standardized evaluations explicit and reproducible.
  • Standardization choices: Answer extraction, invalid-response handling, partial-credit scoring, and log-probability versus generative scoring admit defensible alternatives.The paper treats these as assumptions requiring explicit documentation rather than uniquely determined benchmark specifications.

4 Benchmark-Level Reliability Failures

Across eight cybersecurity benchmarks, 15 recurring failures span all five pipeline stages, from dataset construction through aggregation. These failures can substantially distort scores and rankings through prompt, inference, extraction, scoring, and aggregation choices.

  • Cross-stage overview: 15 recurring failure modes span all five benchmark pipeline stages across eight cybersecurity benchmarks.The audit examines failures in dataset construction, prompting, inference, extraction and scoring, and aggregation.
  • Inference failures: 85.9 pp is the score increase from correcting RedSage-Bench’s newline stop-sequence mismatch for Qwen3.6.Generation continues to the end-of-sequence token, permits the reasoning span to close, and strips that span before extraction.
  • Inference failures: 81.4% accuracy replaces a 0.3% score when SecEval’s GPT-5.4 output budget is raised from five to 16 tokens.The five-token budget violates Azure OpenAI’s 16-token minimum, producing error payloads instead of model answers.
  • Extraction and scoring failures: 99.8 pp separates correct-over-valid and correct-over-total scoring for Gemma-4 on CTI-RCM.Only two of 1,000 predictions are parseable, and both are correct; excluding invalid responses yields 100.0% instead of 0.2%.
  • Extraction and scoring failures: Up to five ranking positions change when CTI-Bench and AthenaBench use different metric directions and scales for similar CVSS-vector tasks.CTI-Bench uses lower-is-better MAD, whereas AthenaBench converts MAD into a higher-is-better normalized percentage.
  • Aggregation failures: Standardized aggregation changes Gemma-4’s CTI-RCM and SecEval scores to 70.9% and 78.3%, respectively.The revised comparison retains invalid predictions and addresses upstream prompt and denominator inconsistencies.

5 Cross-Benchmark Instability

The 23 tasks provide redundant score-level evidence, yet model rankings can disagree across similar tasks and change substantially when pipeline choices are standardized.

  • 5.2 Rank-Level Disagreement: Weak Kendall’s τ-b agreement of 0.29 and 0.24 shows that semantically similar CTI-Bench and AthenaBench tasks rank models differently.The vulnerability-scoring pair also reorders models by up to five positions.
  • 5.3 Ranking Shifts Under Standardization: Nine of 10 models shift by at least three ranks on at least one benchmark after nine pipeline fields are standardized.The harness standardizes fields wherever benchmark semantics permit.
  • 5.3 Ranking Shifts Under Standardization: Spearman’s ρ is −0.47 for MMLU-CS and 0.03 for SecEval, indicating substantial disagreement between original and standardized rankings.Other benchmarks range from 0.42 to 0.73.
  • 5.3 Ranking Shifts Under Standardization: Every rank shift of at least three positions retains the same direction in at least 97.7% of 5,000 paired bootstrap samples.All 95% confidence intervals for Spearman’s ρ exclude 1, so sampling noise alone does not explain the changes.
  • Implications for Benchmark Design: Reliable comparisons require diverse capability coverage together with explicit, consistent evaluation pipelines.Broader task coverage alone does not guarantee informative or stable model comparisons.

6 Toward Reliable Benchmarking

Reliable benchmarking requires documenting the full measurement pipeline, reducing avoidable nondeterminism, and making underspecified evaluation choices explicit and executable.

  • Pipeline Specification: Benchmark releases should specify provenance, prompts, inference settings, extraction, denominators, scoring, aggregation, and reliability statistics.The recommended field list includes invalid-response rates and rank stability.
  • Reproducibility: Reproducibility improves when model configurations and judge settings are pinned and raw outputs are stored for later re-scoring.Stored outputs allow extraction, scoring, denominator, and aggregation changes without querying the evaluated model again.
  • Comparability: Similar benchmarks should align metric direction, scoring rules, alias handling, and aggregation, or report their scores as distinct measurements.Cybersecurity gold labels should also be periodically audited against authoritative sources with uncertainty reported.
  • Manual Judgment: Some reliability failures require manual judgment because benchmark artifacts cannot reliably determine intended partial credit or semantic equivalences.Harnesses should encode these decisions explicitly in executable form.

7 Related Work

Related work establishes that LLM evaluation is sensitive to configurable design choices, while this paper distinguishes measurement reliability in cybersecurity from broader construct validity.

  • Prior Evaluation Studies: Prior studies report sensitivity of LLM performance to prompt wording, decoding parameters, evaluator design, and extraction rules.The cited work motivates auditing evaluation beyond the dataset and metric.
  • Evaluation Comparison: Table 5 reports rank shifts under pipeline standardization, with Spearman’s ρ and bootstrap confidence intervals summarized in its bottom row and Figure 3.Positive and negative Δr_b(m) values indicate upward and downward movement, respectively.
  • Construct Validity and Reliability: Bean et al. study construct validity across benchmark domains but do not include cybersecurity benchmarks.This paper instead focuses on measurement reliability in executable cybersecurity evaluation pipelines.

8 Conclusion

The audit treats cybersecurity benchmarks as end-to-end measurement pipelines and finds that their scores and comparative conclusions depend on pipeline choices and task coverage.

  • 8 Conclusion: Across eight benchmarks, 23 tasks, and 10 LLMs, the audit identifies 15 recurring failure modes spanning all five pipeline stages.The audit examines benchmark reliability as an end-to-end measurement process.
  • 8 Conclusion: Individual pipeline choices can shift scores by more than 80 percentage points, while nine of 10 models shift by at least three ranks under standardization.Broader task coverage can still provide redundant evidence and unstable comparisons.

9 Software

The open-source harness supports model-agnostic cybersecurity LLM evaluation and exports aggregate scores with pipeline metadata for transparency.

  • 9 Software: The harness provides a common evaluation interface for hosted APIs and locally served models.It supports configurable LLM judging for open-ended responses, with evaluated and judge models selected independently.
  • 9 Software: Aggregate scores and pipeline metadata can be exported to EveryEvalEver while question-level content is withheld.The export arrangement follows the paper’s Ethical Considerations.

Limitations

The audit’s conclusions are bounded by its cybersecurity-focused sample, contestable standardization choices, and emphasis on measurement reliability rather than construct validity.

  • Scope: The empirical findings are specific to cybersecurity benchmarks, although some mechanical failures may transfer to other domains without direct testing.Cybersecurity makes failures unusually observable through authoritative ground truth, structured answers, and a heterogeneous benchmark ecosystem.
  • Scope: The audit covers eight benchmarks, 23 tasks, and 10 LLMs, so other systems may exhibit failures outside the taxonomy.Some benchmark artifacts are underspecified, and alternative implementation choices may be defensible.
  • Validity: The study evaluates measurement reliability, not construct validity, so reproducible and internally consistent benchmarks may still miss the real-world capability they claim to represent.The paper characterizes reliability as necessary but not sufficient for valid evaluation.
  • Standardization: Comparative conclusions remain conditional because four standardization choices admit defensible but non-unique alternatives.These choices concern answer extraction, invalid-response denominators, partial credit, and log-probability versus generative scoring.
  • Uncertainty: Question-level bootstrap estimates do not capture variability across repeated model generations.Gold-label and judge verification are also LLM-assisted, with manual validation on random stratified samples.

Ethical Considerations

The paper frames unreliable cybersecurity evaluation as an ethical risk while limiting the work to auditing existing benchmarks rather than developing offensive capabilities.

  • Ethical scope: The work audits benchmark reliability rather than developing new offensive cybersecurity capabilities.It introduces no exploit procedures, malware implementations, offensive datasets, or attack automation.
  • Materials: The audit evaluates existing public benchmarks and authoritative cybersecurity sources covering vulnerabilities, attack techniques, malware behavior, and threat intelligence.The implementation section describes the scored benchmark subsets, task inventory, serving environment, prompts, and configuration records.
  • Ethical concern: Unstable pipelines can support misleading claims about model capability, safety, or specialization that influence selection, deployment, and trust.The paper’s stated goal is to improve transparency, reproducibility, and scientific reliability.
  • Dual use: Detailed failure-mode descriptions could facilitate benchmark-specific optimization or gaming.The paper therefore emphasizes evaluation mechanisms and reports aggregate results and pipeline metadata rather than unnecessary question-level content.
  • Evaluation scope: The scored audit scope comprises 48,662 questions across 23 tasks.Per-task counts sum to this total, while benchmark size discrepancies reflect reported, released, and scored dataset conventions.
  • Evaluation scope: SecBench contributes 661 English MCQs because only 3,000 questions are publicly released and its 270 SAQs are not English.This is a subset of the 47,910 questions reported by the benchmark paper.

C Pipeline Specification Gaps

Pipeline specifications frequently diverge from executable benchmark behavior, and explicit standardization reveals large score changes that depend on evaluation conventions.

  • Specification status: 44 of 72 pipeline configuration fields (61%) cannot be reproduced from benchmark documentation alone.The audit classifies 36 fields as undefined, 8 as contradicted, and 28 as specified and matched.
  • Specification status: Unspecified fields require evaluator choices that can alter prompts, generations, extracted answers, scores, or aggregations without changing questions or models.The audit records such resolutions in a pipeline ledger rather than treating them as benchmark specifications.
  • Contradictions: Documented and released behaviors can conflict, including scoring rules, normalization constants, decoding settings, and prompt templates.These contradictions prevent direct reconstruction of some benchmark metrics or executable settings.
  • Measurement design: Single-field counterfactuals vary one configuration field while holding remaining fields fixed; upstream changes require regeneration, whereas downstream changes can rescore stored outputs.Some interactions cannot be isolated and are reported separately as peer-gap estimates.
  • Standardization: The harness distinguishes documented, released, and standardized layers, treating supplied or normalized settings as evaluation choices rather than uniquely correct interpretations.Four choices admit meaningful alternatives, so measured comparative differences remain conditional on conventions.
  • Measured effects: 90.6 pp separates Primus-Nemotron’s CWET scores under correct-over-valid and correct-over-total denominator policies.The standardized policy retains invalid or unparseable answers in the evaluation population.
  • Measured effects: Qwen3.6’s benchmark mean changes from 0.0% to 85.9% after a pipeline fix.Per-task recovery ranges from 81.4 to 90.1 percentage points.
  • Measured effects: 40.9 pp separates Gemma-4’s scores under one normalization, while Qwen3.6 moves 26.7 pp in the opposite direction.These are benchmark- and model-specific effects of the reported standardization differences.

E.2 Judge Validation

Judge validation finds that the evaluation judge generally parses and grades responses consistently, while targeted checks identify bounded disagreement and label-quality issues. The audit also distinguishes triage signals from benchmark-wide error rates.

  • Extraction and schema validation: 100.0% of multiple-choice judge calls and 99.99% of open-ended calls conform to the requested JSON schema.Fewer than 0.01% require tolerant parsing, and every successfully parsed verdict is CORRECT or INCORRECT.
  • Independent validation: 94.6% of the 1,733 NONE-stratum calls are corroborated by an independent judge, while judges agree on 98.1% of regex-disagreement calls.These targeted strata focus on difficult extractions and verdict disagreements rather than a random sample.
  • Human validation: 94.2% of 120 adversarially sampled questions receive the correct pinned-judge verdict after human adjudication.Annotators achieve Cohen’s κ=0.93, and the seven judge errors include both over-crediting an uncommitted answer and under-crediting a committed answer.
  • Cross-judge agreement: 99.6% of verdicts agree between the pinned and independent judges, with Cohen’s κ=0.99 and a maximum per-model score difference of 0.22 pp.The independent-judge ranking has Spearman ρ=0.98, with GPT-5.4 ranked first under both judges.
  • Gold-label audit: 23.8% is the confirmed-label-error rate among checked disagreement flags, not a benchmark-wide label-error rate.The 1,140 flagged questions are selected through model disagreement, so the study does not extrapolate this rate to the full benchmarks.

G.4.2 Task-level metric drift (F2(A))

Task-level metric conventions can produce divergent model comparisons even for semantically similar cybersecurity tasks. The analysis separates shared performance structure from ranking instability and shows that pipeline changes, especially generation choices, can reorder models.

  • Metric conventions: CTI-Bench’s Correct+Plausible scoring can raise a model’s score by up to 30 pp, while cross-benchmark scoring conventions produce gaps up to 70 pp.CTI-Bench and AthenaBench use different attacker-attribution credit rules, including strict binary versus related-actor credit.
  • Effective dimensionality: 95.25% of task-score variance is explained by the first principal component, indicating that most tasks measure one dominant performance axis.Parallel analysis finds that only this component exceeds the corresponding null eigenvalue.
  • Rank agreement: Kendall’s τ-b is 0.29 for vulnerability scoring and 0.24 for attacker attribution between CTI-Bench and AthenaBench.The vulnerability-scoring comparison has rank displacement of up to five positions despite evaluating the same CVSS quantity.
  • Task similarity: Semantic similarity and ranking agreement are moderately associated across 253 task pairs, with r = 0.516 and p < 0.002.The paper cautions that each τ is estimated from only 10 models, so r² ≈ 0.27 is not treated as a precise variance fraction.
  • Generation versus extraction: Generation changes produce more ranking reordering than extraction on five of eight benchmarks, including AthenaBench with ρgen = 0.34 versus ρext = 0.99.On CTI-Bench, the corresponding correlations are 0.59 and 0.89; MMLU-CS is sensitive to both generation and extraction.
  • Interpretive boundary: The harness automates logging, validation, parsing diagnostics, re-scoring, and aggregate comparisons but does not determine benchmark intent or semantic ground truth.Choices such as partial credit and alias equivalence remain methodological decisions.

J.2 Evaluation Pipeline Card

The evaluation pipeline card records the task-level choices that produce a benchmark score, rather than treating the benchmark as a single fixed specification. For CTI-Bench’s VSP task, it links dataset, prompt, inference, extraction, scoring, aggregation, reliability, and reproducibility details to the reported result.

  • Purpose: An evaluation pipeline card should be provided for each scored task because tasks within one benchmark can use different evaluation choices.These choices include prompts, inference settings, extraction procedures, metrics, denominator policies, and aggregation rules.
  • Card contents: The card should identify the benchmark, task, split, metric, pipeline version, and stable identifiers needed to distinguish related evaluation variants.The proposed machine-readable record uses versioned field names so results link unambiguously to the task, metric, and pipeline configuration.
  • Dataset and metric: For CTI-Bench VSP, all 1,000 released questions use CVE descriptions as inputs and CVSS v3.1 vectors as targets, without a snapshot date for the underlying references.The task reports CVSS mean absolute deviation, with lower values indicating better performance.
  • Prompt, inference, and extraction: The standardized VSP pipeline preserves the task prompt, uses native chat formatting, and applies deterministic inference with a 2,048-token budget, temperature 0, top-p 1.0, and no stop sequence.The extraction policy removes <think>...</think> reasoning and returns a normalized CVSS:3.1/ vector without repairing unstated answers.
  • Reliability: 30.4% disagreement occurs between final-line and anywhere CVSS extraction on cases where at least one rule extracts a vector.The task retains empty or unparseable responses in the attempted-question population and recommends reporting invalid responses alongside the score.
  • Reproducibility: A reproducible run should record the pipeline version, exact prompt and decoding configuration, extraction and scoring rules, denominator policy, and raw responses.Stored responses support rescoring after extraction, scoring, or denominator changes, whereas prompt or inference changes require new generations.
Loading 2609.08765v1…