PaperlayerView exact revision on arXiv

Source-linked AI summary

From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

Pedro Conde, Henrique Branquinho, Valerio Mazzone, Bruno Mendes, André Baptista, Nuno Moniz

arXiv:2605.10834v3cs.AIcs.CR

TL;DR

Existing benchmarks offer limited guidance for selecting AI pentesting agents for realistic targets because they emphasize predefined tasks in simplified settings. This paper proposes a validated vulnerability-discovery protocol for complex targets, showing that finding-level and cumulative evaluation expose trade-offs hidden by binary success metrics.

  • Problem

    Existing task-specific benchmarks in simplified targets do not adequately evaluate discovery of multiple vulnerabilities across complex targets and open-ended exploration.

  • Method

    The paper proposes an adaptable protocol centered on validated vulnerability discovery, semantic matching, bipartite resolution, maintained ground truth, repeated runs, cumulative evaluation, and efficiency metrics.

  • Results

    Finding-level evaluation exposes trade-offs hidden by binary success metrics, while single runs incompletely cover targets and repeated aggregation reveals differing recall, precision, cost, runtime, and consistency behaviors.

  • Takeaways & Limitations

    Evaluating vulnerability coverage, validity, efficiency, and consistency together provides a more operationally informative basis for comparing AI pentesting agents.

  • Takeaways & Limitations

    The protocol contributes no new targets and depends on sufficiently realistic targets containing multiple vulnerabilities across distinct components and attack surfaces.

Abstract

from arXiv · show

AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings. These tools are valuable for measuring bounded capabilities, yet they do not adequately capture the complexity, open-ended exploration, and strategic decision-making required in realistic pentesting. In this paper, we present a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes. The protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, bipartite resolution to score findings under realistic ambiguity, continuous ground-truth maintenance, repeated and cumulative evaluation of stochastic agents, efficiency metrics, and reduced-suite selection for sustainable experimentation. This protocol extends the state of the art by enabling a more realistic, operationally informative comparison of AI pentesting agents. To enable reproducibility, we also release expert-annotated ground truth and code for the proposed evaluation protocol: https://github.com/ethiack/ethibench.

1 Introduction

The paper argues that AI pentesting agents require evaluation methods aligned with real-world use rather than narrow benchmark success. It proposes an adaptable protocol centered on validated vulnerability discovery and operationally meaningful assessment.

  • Motivation: Agentic AI systems combine planning, tool use, iterative reasoning, and environment interaction, enabling adaptation to tasks that are not precisely specified in advance.Pentesting is presented as a clear example of this open-ended task setting.
  • Motivation: The central evaluation challenge is determining whether AI pentesting agents will perform effectively in real-world use rather than merely achieve strict benchmark success.The paper frames this as part of the broader evaluation problem for agentic AI systems.
  • Limitations of existing evaluations: Controlled benchmarks can overstate realistic performance because reproducibility, low scoring cost, and methodological tidiness often depend on simplifying deployment-relevant difficulties.This creates a practical risk that evaluation rewards systems appearing strong under controlled conditions.
  • Limitations of existing evaluations: Existing evaluations mostly use task-specific testbeds with predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity in simplified targets.These formulations remain useful for measuring bounded capabilities but do not fully answer how to assess realistic pentesting performance.
  • Our approach: The paper proposes an adaptable evaluation protocol that supports realistic, operationally meaningful assessments across target classes and intended real-world contexts.Rather than introducing another fixed benchmark, the methodology centers evaluation on validated vulnerability discovery.

2 Related work

Existing AI pentesting evaluations progress from controlled CTF tasks toward more realistic execution-grounded and workflow-based settings, but commonly retain predefined objectives or bounded tasks. This work instead evaluates validated vulnerability discovery in realistic targets by matching agent reports to ground truth and resolving ambiguous findings.

  • 2 Related work: CTF-style benchmarks offer controlled tasks and inexpensive automatic scoring, but reduce success to closed-setting capture-the-flag outcomes that weakly reflect realistic pentesting.Realistic pentesting requires exploration of noisy targets and strategic decisions beyond isolated offensive capabilities.
  • 2 Related work: More realistic workflows introduce multi-service exploitation, distractors, exploit chains, active defenses, or stage-level expert evaluation, yet retain single objectives or predefined flag-based and similarity-based goals.TermiBench uses remote code execution, PACEbench adds environmental complexity, and PentestEval provides fine-grained stage-level evaluation.
  • 2 Related work: Real-vulnerability and execution-grounded benchmarks improve exploit-validation quality, but largely evaluate bounded tasks or predefined attack goals rather than end-to-end pentesting with multiple possible findings.CVE-Bench uses real web CVEs, while CyberGym uses vulnerable-versus-patched execution for stronger validation.
  • 2 Related work: Trajectory-rubric evaluation scores operational objectives, but remains distinct from assessing whether agents discover valid vulnerabilities across realistic targets.PentestJudge evaluates trajectories against a rubric tree of operational objectives.
  • 2 Related work: The proposed methodology shifts evaluation from task completion or trajectory similarity to vulnerability findings, using semantic matching and bipartite resolution to map reports to ground truth.Success is tied neither to flags, a prescribed exploit goal, nor rubric compliance, but to uncovering valid vulnerabilities in realistic targets.

3 A realistic evaluation protocol for AI pentesting agents

The protocol prioritizes realism by evaluating exploration, planning, and strategic decision-making in sufficiently complex targets rather than prescribing a fixed benchmark suite. It combines structured ground truth, semantic matching, bipartite resolution, ongoing validation, repeated evaluation, cumulative analysis, and data-driven reduced suites.

  • Protocol scope: The protocol evaluates agents in sufficiently complex environments requiring exploration, planning, and strategic decision-making, while remaining adaptable across target classes.It prioritizes realism over closed-world benchmark convenience and does not prescribe a fixed target suite.
  • Evaluation procedure: Evaluation proceeds through structured ground-truth construction, LLM-based semantic matching, and maximum bipartite matching to resolve ambiguous correspondences.The matching design permits multiple candidate matches initially, then credits each ground-truth entry at most once and prevents duplicate reports from inflating detections.
  • Ground-truth maintenance: Reliable metrics require periodic expert review of unmatched findings and continuous ground-truth maintenance, followed by automatic re-evaluation with updated entries.Review refines vague or restrictive entries and adds validated vulnerabilities missing from the reference files.
  • Statistical and cumulative evaluation: Low replication is often unavoidable because realistic agent evaluations are computationally and financially expensive, motivating significance tests, effect sizes, and cumulative results alongside per-run comparisons.Welch’s t-test is recommended for pairwise comparisons because stochastic behavior can produce unequal variances and different sample sizes; cumulative results better represent ongoing pentesting.
  • Reduced evaluation suites: Reduced evaluation suites can be selected from historical agreement data using Pearson correlation primarily and Spearman correlation secondarily, but require revalidation and cannot replace periodic full-benchmark assessment.Pearson preserves ranking and performance magnitudes, whereas Spearman measures rank agreement and is less sensitive to outliers or nonlinear scaling.

4 Experiments and results

Experiments apply the protocol to representative agentic pentesting workflows across realistic and CTF-style targets, revealing substantial performance and efficiency variation. Finding-level, repeated evaluation exposes incomplete single-run coverage, accumulation effects, and trade-offs between discovery, precision, noise, cost, and runtime.

  • Experimental setup: The evaluation compares Strix, PentAGI, and Claude Code as representative workflows supporting complex tool use, multi-step reasoning, and agent handoffs.The systems are not intended to establish a definitive ranking because they are highly configurable and extensible.
  • Experimental setup: The experiments use vuln-bank, paygoat, and xben-090, combining realistic vulnerable applications with a CTF-style benchmark target.vuln-bank and paygoat support multi-finding discovery, whereas xben-090 traditionally uses binary success as its criterion.
  • Validation: The findings-to-ground-truth pipeline is checked against 50 expert-triaged findings, including 25 true positives and 25 false positives.The expert annotations are compared with matching performed by different LLMs across three runs.
  • Results: Finding-level evaluation reveals trade-offs among vulnerability discovery, false positives, duplicates, cost, runtime, noise, and practical deployment constraints.A precise system may miss vulnerabilities, while a more exploratory system may generate noisier or more expensive outputs; isolated-run averages can hide these effects.
  • Results: All configurations show substantial false negatives: no single run fully covers the targets or reports every known vulnerability.The comprehensive ground truth is intended to include all known vulnerabilities in the selected targets, while evaluation goals can be adapted.
  • Results: Aggregating three independent runs improves F1-score for every setup, primarily through Recall gains, but the magnitude and behavior of accumulation vary across systems.Two agentic systems using the same LLM can respond very differently to accumulated findings, while relative setup rankings change little.

5 Limitations and future work

The work evaluates an evaluation protocol rather than introducing a new benchmark suite, so its practical usefulness depends on sufficiently realistic targets. Future work should expand realistic environments, study long-term memory under cumulative evaluation, and add safety assessment.

  • Limitations: The work contributes an evaluation protocol rather than new targets, limiting practical usefulness to applications on sufficiently realistic targets.Such targets should contain multiple vulnerabilities across distinct components and attack surfaces.
  • Future work: Future work should create additional realistic targets with broad vulnerability sets, including systems that combine multiple real-world vulnerabilities.
  • Future work: Future work should study multiple forms of long-term memory for agentic pentesting and evaluate their effects under cumulative evaluation settings.
  • Future work: Future work should extend the protocol with safety assessment, including whether agents can be effectively guardrailed.

6 Final remarks

The paper argues that evaluating AI pentesting agents is consequential for deployment in security-critical domains. Its realistic, finding-centered protocol aims to make benchmark performance more operationally informative and relevant to trustworthy real-world use.

  • AI pentesting agents may be deployed in security-critical domains such as financial services and healthcare infrastructure.
  • Benchmarks focused on capture-the-flag or single-goal exploitation can misrepresent the properties required for trustworthy deployment.
  • The proposed evaluation makes assessment more realistic, finding-centered, and operationally informative.

A Main performance metrics · B Additional considerations on evaluating pentesting agents · B.1 Choosing realistic targets

The evaluation framework scores matched pentesting findings with count-, classification-, severity-, and coverage-based metrics, while allowing metrics to reflect operational priorities. It also argues that realistic targets must contain multiple vulnerabilities and plausible attack paths across services or application locations to test exploration, planning, and strategy.

  • A Main performance metrics: After semantic candidate matching and bipartite resolution, the protocol computes TP, FP, and FN from matched, unmatched findings and ground-truth entries.TP counts findings matched to ground truth; FP counts findings without matches; FN counts unmatched ground-truth entries.
  • A Main performance metrics: Recall measures recovered ground-truth vulnerabilities as TP/(TP + FN), while precision measures reported findings corresponding to ground truth as TP/(TP + FP).Higher recall means fewer missed real vulnerabilities, whereas higher precision means fewer spurious or incorrect findings.
  • A Main performance metrics: F1 summarizes precision and recall with equal importance, whereas F0.5 weights precision more heavily than recall.F0.5 is useful when reducing false positives matters more than maximizing coverage.
  • A Main performance metrics: CWE coverage counts distinct CWE categories among recovered true positives, indicating whether an agent detects varied vulnerability types rather than a narrow subset.Duplicates are findings with candidate matches that are excluded from the final one-to-one assignment and may represent repeated reports of one vulnerability.
  • A Main performance metrics: Severity scoring sums points for true-positive ground-truth entries, distinguishing recovery of high-impact vulnerabilities from detection of lower-severity issues.The example mapping assigns 3 points to severity 0.1–3.9, 15 to 4.0–6.9, 30 to 7.0–8.9, and 50 to 9.0–10.0.
  • A Main performance metrics: Metric selection is flexible: the structured correspondence supports derived or constrained comparisons, such as recall subject to a minimum precision threshold of 95%.The choice should align with the evaluator’s goals and operational setting.
  • B.1 Choosing realistic targets: Realistic evaluation targets should include multiple vulnerabilities distributed across services, components, or application locations, rather than a single flaw or constrained attack path.Such targets test exploration, alternative-opportunity reasoning, prioritization, and decisions about how opportunities relate to broader objectives.

B.2 Training-data contamination · B.3 Temporal variance in model behavior

The evaluation protocol must account for training-data contamination and temporal variance in remotely hosted model behavior. These factors can distort measured agent capabilities and complicate fair comparisons.

  • B.2 Training-data contamination: Publicly available targets may be contaminated by the underlying model’s training data.Contamination may involve the target, its write-up, or closely related exploit material.
  • B.2 Training-data contamination: Contamination can overestimate an agent’s ability to explore, reason, and discover vulnerabilities.This risk is especially relevant when evaluating widely circulated benchmark instances.
  • B.2 Training-data contamination: Target selection and result interpretation should explicitly consider training-data contamination.The concern applies when publicly available evaluation material may have entered model training data.
  • B.3 Temporal variance in model behavior: Remotely hosted LLM evaluations may vary over time because of provider-side updates and alias remapping.These changes are outside the evaluator’s control and can alter measured performance.
  • B.3 Temporal variance in model behavior: Load-dependent degradation and other operational factors can also change API-based model behavior.Observed performance differences may therefore reflect service conditions rather than agent capabilities.
  • B.3 Temporal variance in model behavior: A/B comparisons should run in parallel or within the narrowest possible time window.This reduces the likelihood that observed differences reflect changes in the underlying hosted service.

C Additional considerations on the experimental setup

The experiments used repeated black-box pentests in clean, containerized environments with standardized inputs and infrastructure. Outputs were normalized to finding-level JSONL when necessary to support the evaluation protocol.

  • Experimental procedure: Each engine–model combination completed three headless black-box runs per target in a freshly installed, clean containerized environment.Only the target URL was provided as target-specific input, preventing information reuse across repeated pentests.
  • Prompting and outputs: Agents received a common prompt to find and exploit all target vulnerabilities while maximizing application coverage and completing the pentest.Claude Code additionally received cybersecurity-expert role instructions and was required to produce JSONL findings with specified fields and discovery timestamps.
  • Execution infrastructure: Experiments ran on an e2-standard-4 Google Cloud VM with 4 vCPUs, 16 GB memory, and six parallel runs.Most compute was attributed to pentesting-tool overhead, with remaining computation delegated to LLMs through API calls.
  • Output normalization: PentAGI’s final reports were post-processed into itemized JSONL findings, whereas Strix and Claude Code already produced protocol-compatible itemized findings.PentAGI’s conversion used a multi-turn AI-as-judge process.
  • Model standardization: PentAGI’s agents used the same model under each experimental condition despite supporting fine-grained model selection across agent roles.This standardization avoided assigning different models to PentAGI agents based on task complexity.

D Additional considerations on finding-to-ground-truth sanity-check

The sanity-check evaluates finding-to-ground-truth matching with cost-efficient LLMs and repeated experiments, while accounting for stochasticity and duplicate labels. The matching prompt compares vulnerability meaning across structured finding and ground-truth descriptions rather than relying solely on wording.

  • Experimental setup: Four relatively small, cost-efficient LLMs were used at temperature 0.3 because pipeline cost scales with the number of finding–ground-truth pairs.The selected models were Deepseek 3.1, Gemini 3 Flash, GPT 5.4 Mini, and Haiku 4.5; larger models would add computational and monetary burden.
  • Experimental setup: Each experiment uses 3 replicates, with mean and standard deviation recorded; consistently low standard deviations indicate minimal stochasticity across runs.Repeated evaluation supports reporting both central tendency and run-to-run variation.
  • Scoring interpretation: Bipartite resolution can classify findings as duplicates, true positives, or false positives, so the sum of TPs and FPs does not always equal 50.Duplicate labeling explains why these two outcome counts may not cover all findings.
  • LLM-as-judge matching: The matching prompt asks whether an agent finding and ground-truth entry describe the same vulnerability despite potentially different wording.It instructs the judge not to determine equivalence solely from textual wording.
  • LLM-as-judge matching: The prompt supplies the finding’s name, description, and reproduction steps alongside the ground truth’s name, category, description, and additional information.These fields provide structured context for semantic comparison.

E Additional considerations on agentic pentesting results · F Additional experiments and results

The results show that pentesting-agent performance depends on both engine–model configuration and operational priorities. Claude Code leads overall, while PentAGI and Strix offer distinct trade-offs among coverage, severity, false positives, cost, and scan duration.

  • E Additional considerations on agentic pentesting results: Claude Code achieves the strongest overall performance and efficiency, with high true-findings volume, highest F-scores, broadest CWE coverage, and lowest cost and time.Its results also include high-impact vulnerability discovery, reflected in the severity metric.
  • E Additional considerations on agentic pentesting results: PentAGI achieves the second-best overall F-score and highest severity score, but incurs higher cost, longer scans, and the largest false-positive count.This configuration prioritizes severity and performance at the expense of operational efficiency and precision.
  • E Additional considerations on agentic pentesting results: Strix reports fewer false positives than other engines but finds fewer vulnerabilities, producing cleaner outputs with reduced coverage.Its result profile represents the opposite trade-off from PentAGI’s higher severity and broader but noisier reporting.
  • E Additional considerations on agentic pentesting results: The underlying model materially changes engine behavior, with open-source models generally reporting fewer findings, more false negatives, and weaker recall.Fewer reported findings can sometimes improve precision, but the passage associates open-source models with reduced recall.
  • E Additional considerations on agentic pentesting results: Claude Sonnet 4.6 is the best-performing model across the evaluated setups.The passage emphasizes that the same engine can behave very differently when paired with different model backends.
  • E Additional considerations on agentic pentesting results: The protocol supports practical comparison by exposing trade-offs beyond F-score, including discovery rate, false positives, duplicate reporting, runtime, and cost.A setup that leads overall may still require scrutiny for deployment-specific constraints; Claude Code’s 81.24% precision is given as an example.

F.1 Per-target results · F.2 Temporal evaluations · F.3 Pairwise statistical comparisons

The evaluation reveals substantial target- and run-dependent variation in vulnerability discovery, which global metrics alone can obscure. Temporal and pairwise analyses further expose discovery dynamics, statistical uncertainty, and distinct precision–recall strategies.

  • F.1 Per-target results: xben-090 stresses output precision: setups reporting relatively few findings can still show high false-positive rates because reported issues often lack validation.The target therefore emphasizes finding quality rather than finding volume.
  • F.1 Per-target results: vuln-bank contains 60 ground-truth vulnerabilities, making it useful for breadth and CWE-coverage analysis even when high-performing setups achieve only partial single-run recall.Its size and category distribution support coverage-oriented evaluation.
  • F.1 Per-target results: Across three paygoat runs, setup differences converge, with Strix-Qwen showing substantial Recall improvement cumulatively despite moderate per-run performance.This pattern indicates that stochasticity in some engine–model combinations can benefit cumulative evaluation.
  • F.1 Per-target results: PentAGI-Sonnet’s cumulative F1 decreases relative to its per-run average because false positives accumulate disproportionately, reducing Precision more than Recall gains improve it.Cumulative evaluation can therefore penalize setups whose additional findings are mostly invalid.
  • F.1 Per-target results: Per-target evaluation is essential because overall metrics can obscure behavioral differences and system rankings may shift with target complexity, structure, and vulnerability distribution.Independent target analysis enables finer-grained assessment of capabilities under different stress conditions.
  • F.2 Temporal evaluations: Temporal analysis tracks true and false positives, severity, and CWE coverage, revealing diminishing returns that aggregate metrics alone cannot show.For Claude Code, true positives, severity, and CWE coverage grow roughly quasi-linearly, while paygoat false positives can accelerate as true-positive growth plateaus.
  • F.3 Pairwise statistical comparisons: Pairwise comparisons show that low replicate counts can make Welch’s t-test underpowered, so nonsignificant p-values may coexist with large effects, including a +12.35% F1 difference and Cohen’s d = 2.168.The pentagi-sonnet versus strix-sonnet comparison has p = 0.0577, just outside the conventional threshold.
Loading 2605.10834v3…