PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Paul Kassianik, Blaine Nelson, Yaron Singer

arXiv:2607.15263v3cs.CRcs.AI

TL;DR

Security-agent evaluations often emphasize peak offensive capability under generous budgets, leaving cost-constrained operational usefulness less directly characterized. This paper compares offensive Cybench and defensive BOTS v1 agents at fixed cost levels, finding that extra compute improves some CTF performance while SOC success depends more on disciplined tool use than raw budget alone.

  • Problem

    Security-agent evaluations emphasize peak offensive capability under generous inference budgets, providing incomplete evidence about cost-constrained operational security performance.

  • Method

    The paper evaluates offensive Cybench and defensive BOTS v1 agents in a common harness using fixed-cost operating points that separate inference and tool spending.

  • Results

    Offensive CTF performance shows headroom with additional budget, whereas BOTS v1 gains less from extra budget and often associates higher tool volume with lower scores.

  • Takeaways & Limitations

    Security-agent evaluation should be cost-aware and workflow-specific because offensive CTFs and defensive SOC investigations exhibit different scaling patterns.

  • Takeaways & Limitations

    The defensive analysis covers only 31 scored BOTS v1 questions from a public Splunk dataset, so its SOC conclusion is not a final claim about production readiness.

Abstract

from arXiv · show

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.

1. Introduction

This paper evaluates offensive and defensive security agents on a common cost-success axis using Cybench and Splunk BOTS v1. It finds that offensive performance benefits from additional reasoning budget, whereas defensive performance depends more on disciplined tool use, telemetry navigation, and selective enrichment.

  • Cost-success evaluation: The evaluation compares model operating points at fixed inference and tooling costs rather than reporting only peak success under generous budgets.For offensive CTFs, marginal spend is mainly test-time reasoning and command execution; SOC investigations also consume Splunk queries, event inspection, and web-search resources.
  • Cost-success evaluation: The common harness evaluates offensive Cybench challenges and defensive Splunk BOTS v1 investigations with task-specific access, scoring, and cost constraints.Cybench supplies sandboxed CTF tasks with shell and Python access, while BOTS v1 supplies public Splunk telemetry, competition questions, point values, and hint penalties.
  • Scaling regimes: Offensive Cybench success shows clear budget headroom for Claude Opus 4.8 and DeepSeek v4 Flash, while GPT-5.5 solves most successful challenges below the lower cap.The study conducts a paired budget-headroom analysis to identify where additional per-sample spending changes observed scores.
  • Scaling regimes: Defensive BOTS v1 success is less predicted by extra budget: increasing DeepSeek v4 Flash’s direct cap yields little gain, and high tool volume often accompanies lower scores.The defensive signal emphasizes disciplined tool use, telemetry navigation, and selective enrichment rather than raw reasoning budget alone.
  • Limitations and accounting: No-tools controls reveal substantial model-dependent direct-answer recovery on public BOTS v1, requiring decontamination checks before interpreting absolute scores.The work also accounts for model-token and priced-tool costs, including cost per 1,000 BOTS points.

2. Related Work

Related work spans broad cybersecurity capability suites, offensive agent benchmarks, SOC investigation evaluations, and benchmark-integrity research. This study extends defensive evaluation by combining official BOTS scoring with fixed-budget, cost, and tool-efficiency analyses while addressing leakage concerns.

  • Cybersecurity evaluation landscape: Cybersecurity evaluation covers knowledge tests, offensive and defensive agent tasks, security risks, and benchmark integrity.CyberSecEval 2 measures prompt injection, code-interpreter abuse, false refusals, and exploit generation, while CyberSecEval 3 extends toward automated social engineering and autonomous offensive operations.
  • Offensive agent benchmarks: Offensive benchmarks increasingly combine interactive execution, tool use, sandboxed commands, and professional capture-the-flag tasks.InterCode, InterCode-CTF, NYU CTF Bench, and Cybench represent this progression; CVE-Bench moves toward real-world web-application exploitation.
  • SOC investigation evaluation: SOC evaluation emphasizes detection, analysis, evidence handling, response coordination, and operational judgment rather than isolated scores or tool-call counts.BOTS datasets provide public telemetry, questions, answers, and scoring artifacts, while investigations require schema discovery, SPL revision, event inspection, artifact correlation, and selective enrichment.
  • Defensive-agent benchmarks: Emerging defensive-agent benchmarks target multi-step incident analysis, graph-linked threat-investigation questions, and embodied threat-hunting workflows.SIABench covers alert triage and deeper workflows across malware, phishing, forensic, and log-analysis scenarios; ExCyTIn-Bench and CyberTeam address graph-linked investigations and function-calling workflows.
  • Cost-aware evaluation and integrity: This BOTS v1 study preserves official scoring while adding fixed-budget comparisons, model-token cost, priced external-tool spend, and tool-efficiency views.Public benchmark use also motivates leakage controls, including protected releases, training-exclusion controls, private or retroactive holdouts, and caution with API-mediated leakage.

3. Evaluation Design

The evaluation uses a ReAct-style agent with automatic context compaction and varies per-sample cost caps to measure success and cost efficiency. It compares offensive Cybench challenges with sequential defensive BOTS v1 investigations, using distinct tool suites and benchmark-specific scoring and refusal rules.

  • Evaluation framework: All benchmark runs use a ReAct-style agent with auto-compaction at 90% of the model context window, changing only the available tools between offensive and defensive tasks.Inspect uses provider-native compaction when available and falls back to summary-based compaction.
  • Offensive benchmark: Cybench hard evaluates sandboxed offensive challenges with bash, Python, and submission tools, allowing three attempts and scoring case-insensitive reference-flag matches across three epochs.The full Cybench set contains 39 challenges spanning cryptography, web exploitation, reverse engineering, forensics, and exploitation.
  • Defensive benchmark: BOTS v1 evaluates 31 scored sequential investigation questions worth 10,300 official points, with Splunk as the primary evidence source and limited external enrichment tools.Agents receive discovery, search, and event-inspection tools over Splunk Search Processing Language, plus VirusTotal, WHOIS, DNS, Brave Search, bash, and Python.
  • Defensive benchmark: The primary BOTS metric is bots_points with hint penalties, while binary includes accuracy is secondary; three-epoch means are reduced to the single-epoch 10,300-point scale.Prerequisite context from earlier official questions is provided for 23 of the 31 scored questions, making the task closer to follow-up analysis than cold-start reconstruction.
  • Cost accounting: Cost caps abort over-budget sample runs as failures, and reported spend separates model inference from priced external enrichment while excluding infrastructure and analyst-review costs.Brave Search costs $0.005 per request, WHOIS History preview costs $0.0258, and WHOIS History purchase costs $1.29; VirusTotal, DNS, and live WHOIS/RDAP are zero marginal cost.

4. Evaluation Results

Evaluation results show a sharp benchmark split: GPT-5.5 is strongest on Cybench, while Claude Opus 4.8 leads BOTS v1 in both defensive score and cost efficiency. Refusal behavior materially shapes Cybench outcomes, whereas BOTS refusals generally do not reduce earned points, and post-verification GPT-5.6 results are observational rather than causal.

  • Cybench: GPT-5.5 solves 94.1% of Cybench hard challenges at $1.16 per solved-equivalent challenge, the strongest operating point observed.Among GPT-5.6 variants, Luna reaches 79.5%, Terra 65.8%, and Sol 9.4% after refusing 90.6% of sample-epochs.
  • BOTS v1: Claude Opus 4.8 leads BOTS v1 with 9,666.7 of 10,300 points (93.9%) and $2.98 per 1,000 points.GPT-5.6 Terra and Sol score 92.1% and 91.4%, Fable 88.4%, and Luna 83.7%; all four exceed the standard GPT-5.5 row.
  • Post-verification comparison: Post-verification GPT-5.6 changes are observational rather than controlled causal interventions: Sol rises from 9.4% to 87.2% on Cybench while falling from 91.4% to 78.8% on BOTS v1.Account status, run date, provider backend behavior, and other temporal effects may differ, and the results should be read alongside contamination controls.
  • Refusal context: Cybench refusals count as failures, with Terra recording 39 failed refusal outcomes, Sol 106, and Fable all 117.Fable’s 0% result reflects refusal of all 117 sample-epochs before any tool call rather than zero underlying capability.

5. Decontamination Controls

Public, aging SOC benchmarks require decontamination controls because no-tools performance is strongly model-dependent and prerequisite context can substantially raise scores. BOTS remains useful as a reproducible harness and cost-accounting testbed, but not as a clean leaderboard alone.

  • Decontamination protocol: The evaluation uses single-epoch no-tools probes that remove Splunk, web, bash, Python, and other tools, testing answers from question text, prerequisites, and prior knowledge.These probes address contamination risks from BOTS v1’s public and aging benchmark data.
  • No-tools results: Without prerequisite context, GPT-5.6 Sol scores 5,200/10,300 points (50.5%), GPT-5.5 scores 54.9%, and Claude Opus 4.8 scores 50.0%, all with zero non-submit tool events.Other models score 13.1% for GPT-5.6 Luna, 18.4% for Terra, and 14.6% for Claude Fable 5, showing substantial model dependence.
  • No-tools results: Official prerequisite Q&A raises every no-tools score, most sharply for Sol from 50.5% to 77.2% and Claude Opus 4.8 from 50.0% to 74.8%.The result demonstrates that prerequisite context materially affects apparent benchmark performance.
  • Interpretation and controls: No-tools, perturbed-question, answer-before-query, private-holdout, or fresh-incident checks should accompany public SOC benchmarks before absolute scores are interpreted as live investigation skill.BOTS remains useful as a reproducible harness and cost-accounting testbed, not as a clean leaderboard by itself.

6. Scaling Findings

Scaling behaves differently across the two security-agent workloads: extra inference budget can materially improve Cybench performance, while BOTS v1 success depends more on efficient, disciplined tool use than on spending alone. Retrospective paired replay under a $0.80 per-sample cap isolates these effects without comparing unrelated traces.

  • Method: Paired retrospective replay caps completed traces at $0.80 per sample, applying model-call spend to Cybench and model-token plus priced-tool spend to BOTS v1.A sample-epoch is failed or zeroed once cumulative cost crosses the cap, making these descriptive replay analyses rather than new benchmark runs.
  • Cybench: 18.8 percentage points: Claude Opus 4.8 gains this much on Cybench beyond the $0.80 cap, versus 10.3 points for DeepSeek v4 Flash and 2.6 points for GPT-5.5.The Claude Opus 4.8 and DeepSeek v4 Flash bootstrap intervals are clearly positive; GPT-5.5 usually solves its challenges below $0.80.
  • BOTS v1: Claude Opus 4.8 captures most BOTS v1 points below $0.80 with only 603 non-submit tool calls, while GPT-5.5 spends almost twice as much and makes more than twice as many tool calls yet trails it.The high-effort GPT-5.5 row does not materially change this comparison.
  • Interpretation: On Cybench, extra budget typically buys longer sandbox exploration, more command execution, and more chances to recover from false starts.On BOTS v1, additional spend often buys Splunk or enrichment calls after the agent has already missed a key field, prerequisite, or validation step.
  • Interpretation: Defensive scaling should measure evidence quality alongside dollars and tokens: finding the right telemetry, validating answers, and using enrichment only when it changes the investigation.The BOTS v1 findings indicate that operationally useful scaling is not captured by spend alone.

7. Limitations

The study’s limitations include observational, non-invariant evaluation conditions and a narrow defensive scope. Accordingly, the authors emphasize fixed-budget operating points rather than universal rankings and avoid overgeneralizing beyond BOTS v1.

  • Evaluation design: Experiments are observational, combining prospectively fixed cost caps with retrospective caps applied to completed traces.The authors therefore emphasize fixed-budget operating points over universal model rankings.
  • Evaluation design: GPT-5.6 comparisons are confounded by changed OpenAI account verification status, which Inspect logs do not encode.The authors report both operating points and avoid attributing differences solely to verification.
  • Evaluation design: MiniMax M3’s six provider/API failures are retained as Cybench failures, exposing a system-level reliability effect.This operational failure mode affects the observed results independently of task-solving capability.
  • Defensive scope: Defensive analysis covers only 31 scored BOTS v1 Po1s0n1vy/Cerber questions from a public Splunk dataset, excluding the official warm-up.BOTS v2 and v3 are broader, covering 51 and 58 questions respectively, but comparable multi-model results are not yet available.

8. Artifact and Ethical Considerations

The study uses public cybersecurity benchmarks in sandboxed environments and reports only aggregate metrics and methodology. It withholds secrets, API keys, and operational exploit instructions.

  • The evaluation uses public cybersecurity benchmarks and sandboxed environments.
  • The paper reports aggregate metrics and methodology only.
  • The authors do not publish secrets, API keys, or operational exploit instructions.

9. Conclusion · Appendix A. Evaluation Run Dates · Appendix B. Uncertainty Method

The paper concludes that security-agent evaluation should be cost-aware and workflow-specific because offensive CTFs and defensive SOC investigations exhibit different scaling patterns and operational requirements. Appendices document evaluation timestamps and descriptive bootstrap procedures for uncertainty estimation.

  • 9. Conclusion: Security-agent evaluation should be cost-aware and workflow-specific because offensive CTFs and defensive SOC investigations stress different behaviors and scale differently.The conclusion distinguishes offensive and defensive evaluation regimes rather than treating model performance as task-independent.
  • 9. Conclusion: More test-time compute can improve CTF success for some frontier and shared models under retrospective caps.The conclusion specifically names Claude Opus 4.8 and DeepSeek v4 Flash as examples.
  • 9. Conclusion: Policy refusals can dominate offensive scores in the new GPT-5.6 and Fable runs.This finding is presented as a qualification on interpreting offensive benchmark results.
  • 9. Conclusion: SOC work does not scale with raw budget in the same way, and success depends on disciplined tool use, telemetry navigation, and selective enrichment.GPT-5.6 Terra and Sol approach the leading Claude Opus 4.8 score, according to the conclusion.
  • Appendix A. Evaluation Run Dates: Appendix A records UTC creation timestamps from source Inspect logs for rows used in the paper’s result, scaling, and robustness tables.Rows appearing in multiple tables are listed once, and retry timestamps are included when retries contribute to reported cost.
  • Appendix A. Evaluation Run Dates: Retrospective cap rows derive from listed source traces rather than separate evaluation runs.This distinction clarifies how capped comparisons relate to the recorded evaluation dates.
  • Appendix B. Uncertainty Method: Appendix B reports descriptive bootstrap intervals by resampling task IDs with replacement while retaining the relevant epochs and scoring weights.Cybench resamples challenge IDs; BOTS v1 resamples question IDs and retains three epochs plus official point weights.
  • Appendix B. Uncertainty Method: Retrospective cap contrasts compare capped and uncapped values for the same sampled task IDs, while paired contrasts resample only IDs common to both runs.The stated procedure applies task-level resampling to preserve the pairing structure of these comparisons.

Appendix C. Cybench Robustness

Appendix C reports Cybench challenge-level bootstrap intervals and clarifies that the DeepSeek v4 Flash $0.80 result uses the Table 2 retrospective cap. Independent recomputation and rounding explain last-digit differences, while ranking contrasts favor GPT-5.5 and higher-budget DeepSeek v4 Flash over Claude Opus 4.8.

  • Bootstrap robustness: Table 9 reports Cybench challenge-level bootstrap intervals, with DeepSeek v4 Flash $0.80 using the same retrospective cap as Table 2.Last-digit differences from the main table reflect independent recomputation and rounding from logs.
  • Model-ranking contrasts: GPT-5.5 exceeds the higher-budget DeepSeek v4 Flash row by 7.7 percentage points, with an interval of +1.7 to +15.4 percentage points.Claude Opus 4.8 does not close the gap to those leaders.
  • Model-ranking contrasts: GPT-5.5 beats Claude Opus 4.8 by +19.7 percentage points [+9.4, +30.8], while higher-budget DeepSeek v4 Flash beats Claude by +12.0 percentage points.These are paired contrasts reported for model-ranking context.

Appendix D. BOTS v1 Robustness · Appendix E. BOTS v1 Sequential-Context Check

Appendix D finds Claude Opus 4.8’s advantage over GPT-5.5 robust across paired bootstrap intervals, while the DeepSeek budget comparison is inconclusive. Appendix E shows that performance remains high under sequential context, with Claude outperforming GPT-5.5 especially on dependent questions and losing fewer points to misses and hint penalties.

  • Appendix D. BOTS v1 Robustness: Table 10 reports question-level bootstrap intervals for BOTS v1 results.These intervals provide the robustness analysis for the reported model margins.
  • Appendix D. BOTS v1 Robustness: 12.8 percentage points is Claude Opus 4.8’s paired margin over standard GPT-5.5, with a bootstrap interval of +0.9 to +27.4 percentage points.Its margin over GPT-5.5 high effort is 12.5 percentage points, with interval +2.4 to +23.7 percentage points.
  • Appendix D. BOTS v1 Robustness: 0.9 percentage points is the DeepSeek v4 Flash $4.20-cap row’s advantage over the $2.10-cap row, with interval −4.9 to +6.9 percentage points.The passage characterizes this budget change as inconclusive.
  • Appendix E. BOTS v1 Sequential-Context Check: 31 scored questions remain after excluding the official warm-up, including 23 with explicit dependencies that receive prerequisite context in the prompt.The evaluation treats these dependent questions as follow-up investigation with known case state rather than cold-start reconstruction.
  • Appendix E. BOTS v1 Sequential-Context Check: Table 11 lists the benchmark metadata’s direct prerequisite chart for dependent scored questions, omitting independent questions and the warm-up.The prompt preamble expands prerequisite edges transitively.
  • Appendix E. BOTS v1 Sequential-Context Check: 96.3% is Claude Opus 4.8’s independent-question score versus 89.7% for standard GPT-5.5, while dependent-question scores are 93.4% versus 79.5%.Claude loses 133.3 points to hint penalties and 500 to misses, compared with GPT-5.5’s 205 and 1,750 points, respectively.
Loading 2607.15263v3…