PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Risk Assessments for Offensive Cybersecurity Agents

Boyi Wei, Benedikt Stroebl, Jiacen Xu, Joie Zhang, Zhou Li, Peter Henderson

arXiv:2505.18384v5cs.CRcs.AI

TL;DR

Most cybersecurity-agent audits use static evaluations, leaving adversarial modification underexplored. The paper assesses agent improvement across degrees of freedom under bounded compute and internal feedback, finding gains above 40% on InterCode CTF with an 8-GPU-hour budget.

  • Problem

    Most evaluations of offensive cybersecurity agents are static and do not consider adversaries using compute to modify agent systems.

    Source: §1 · p. 2

  • Method

    The paper evaluates adversarial agent modifications across degrees of freedom under a fixed compute budget, distinguishing adaptation-time from deployment-time compute.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    More than 40%: InterCode CTF performance can increase relative to baseline with an 8-GPU-hour budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dynamic risk assessments should account for agent modifications because capabilities can improve without expert knowledge, posing risk even from less-sophisticated adversaries.

    Source: §6 · p. 10

    The tested modifications cover only part of the possibilities; real-world adversaries may use browsing tools, manual prompt refinement, RL self-training, or better exploration.

    Source: §6 · p. 9

Abstract

from arXiv · show

Foundation models are increasingly becoming better autonomous programmers, raising the prospect that they could also automate dangerous offensive cyber-operations. Current frontier model audits probe the cybersecurity risks of such agents, but most fail to account for the degrees of freedom available to adversaries in the real world. In particular, with strong verifiers and financial incentives, agents for offensive cybersecurity are amenable to iterative improvement by would-be adversaries. We argue that assessments should take into account an expanded threat model in the context of cybersecurity, emphasizing the varying degrees of freedom that an adversary may possess in stateful and non-stateful environments within a fixed compute budget. We show that even with a relatively small compute budget (8 H100 GPU Hours in our study), adversaries can improve an agent's cybersecurity capability on InterCode CTF by more than 40\% relative to the baseline -- without any external assistance. These results highlight the need to evaluate agents' cybersecurity risk in a dynamic manner, painting a more representative picture of risk.

1 Introduction

Most cybersecurity-agent evaluations are static, although adversaries can modify both models and agent scaffolding. The paper proposes dynamic assessment of those modifications and reports gains across multiple degrees of freedom.

1 Introduction

  • Static evaluations omit adversaries’ use of compute to modify agents, including their scaffolding, tools, planning, and task decomposition.The paper notes that strong verifiers can support self-improvement without external assistance.Source: §1 · pp. 1–2
  • The paper frames dynamic cybersecurity risk assessment as underexplored and studies autonomous agent improvement without external assistance.It motivates the threat model partly through policy discussions about foreseeable modifications and compute thresholds.Source: §1 · p. 2
  • More than 40%: InterCode CTF performance can increase relative to baseline under an 8-GPU-hour budget.The threat model examines how far adversaries can push along modification axes on InterCode CTF (Test).Source: §1 · p. 2, Figure 1 · p. 2
  • The work outlines five degrees of freedom, tests improvement across three CTF benchmarks, and analyzes changes under a fixed compute budget.Table 1 contrasts this dynamic analysis with prior work.Source: §1 · p. 2, Table 1 · p. 2

2 Cybersecurity is Amenable to Self-Improvement

Cybersecurity combines verifiable outcomes and financial incentives that can support compute-driven agent improvement. Whether attackers can repeat attempts depends on whether the environment preserves prior changes.

2 Cybersecurity is Amenable to Self-Improvement

  • Cybersecurity is amenable to self-improvement because compute scaling and non-statefulness let adversaries use more resources and more effective search strategies.The section identifies these as two preconditions for iterative improvement.Source: §2 · pp. 2–3

2.1 Preconditions for compute scaling

  • Strong verifier signals and financial incentives together make self-improvement in real-world attack environments viable.A verifier indicates when a vulnerability is found, while financial incentives motivate allocating computational resources.Source: §2.1 · p. 3

2.2 Stateful and Non-Stateful Environments

  • Stateful environments retain changes and constrain repeated sampling, whereas non-stateful environments can be reset for extensive attempts.Consequently, stateful attackers must improve on held-out data before a one-shot attempt, while repeated sampling can improve performance in non-stateful tasks.Source: §2.2 · pp. 3–4

3 Threat Model and Degrees of Freedom

The threat model bounds adversaries to finite compute and internal environmental feedback, distinguishing offline agent improvement from online task-solving compute. It treats modification of agent components and iterative deployment strategies as separate degrees of freedom.

3 Threat Model and Degrees of Freedom

  • The threat model limits adversaries to bounded compute and internal environment feedback, excluding external assistance.It distinguishes offline adaptation-time compute from online deployment-time compute; the former can support broader transferable capabilities, while the latter improves task-specific capabilities.Source: §3 · p. 4

3.1 Degrees of Freedom

  • Repeated sampling extends the number of trajectories in non-stateful settings, while increasing maximum interaction rounds allows more feedback and attempts within a trajectory.Repeated sampling depends on sample diversity; simply increasing interaction rounds may not fix repeated commands, and stateful environments prevent reverting to prior states.Source: §3.1 · pp. 4–5
  • Iterative prompt refinement changes the initial prompt using failed trajectories and prior refinements, enabling improvement without external assistance.This strategy modifies the initial stage of the agent’s trajectory.Source: §3.1 · p. 5
  • Self-training uses verifier-selected successful solutions from a development set to fine-tune the core model using adaptation-time compute.The approach relies exclusively on internal feedback rather than external hints or corrections.Source: §3.1 · p. 5
  • Iterative workflow refinement uses development-set performance and prior workflows to generate improved agent scaffolding before deployment.The workflow encompasses the end-to-end sequence from input to final action, including reasoning, planning, and tool use.Source: §3.1 · p. 5

4 Experiment Results

The experiments test individual agent-improvement strategies and compare them under fixed compute budgets across stateful and non-stateful settings. They assess how these strategies change offensive cybersecurity performance.

4.1 Repeated Sampling and Increasing Max Rounds of Interactions

  • Increasing repeated samples k and interaction rounds N improves pass@k, but gains diminish as either parameter grows.Larger N can also encounter repetitive command loops and context-window limits, particularly without memory truncation.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 Iterative Prompt Refinement

  • Iterative prompt refinement achieves higher pass@k than repeated sampling by learning from failed trajectories, with gains from one additional inference step.The refinement changes the initial user prompt; unlike repeated sampling, its outcomes are not independent across iterations.Source: §4.2 · pp. 6–7, Figure 3 · p. 6

4.3 Self-Training

  • Fine-tuning on 33 successful trajectories for five epochs generalizes to the InterCode CTF test set, outperforming the base model across k values.Ten epochs can improve development-set pass@1 while reducing output diversity and larger-k pass@k on both development and test sets.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Iterative Workflow Refinement

  • Iterative workflow refinement finds workflows with higher pass@1 than the original, even when the meta-agent and offensive agent share a core model.The weak development-set pass@1 reward leaves substantial variance in intermediate search performance despite a general upward trend.Source: §4.4 · pp. 7–8, Figure 5 · p. 7

4.5 Comparative Analysis Under Fixed Compute Budget

  • More than 40% capability gain was achievable for <$36 of compute, particularly in non-stateful environments.The fixed-budget comparison selects the best-performing configuration for each degree of freedom.Source: §4.5 · p. 8, Figure 6 · p. 8
  • Under limited compute, pre-deployment manipulations are intensive, while inference-time manipulations become the primary risk in non-stateful settings.Pre-deployment methods require a development set and repeated hyperparameter searches and evaluations; the distinction motivates separate risk assessments.Source: §4.5 · pp. 8–9

5 Related Work

Prior work evaluates cybersecurity capabilities through benchmarks, self-improving models, and language agents. This work instead addresses gaps in cybersecurity risk-evaluation methodology rather than proposing a new benchmark.

5 Related Work

  • Agent-based CTF benchmarks approximate real-world conditions, while this work addresses risk-evaluation methodology rather than proposing a new dataset.Earlier evaluations include MCQ and single-turn tasks, while newer frameworks add interactive simulations and structured attack-chain analyses.Source: §5 · p. 9
  • Verifier-guided self-training can support self-correction, and inference-time methods include sampling, search, and iterative refinement; cybersecurity tasks provide built-in verifiers.Workflow-level methods also search over agent-system designs.Source: §5 · p. 9
  • Offensive cybersecurity agents have been studied in penetration testing, exploitation, patching, and CTF solving; this work focuses on single-agent self-improvement to isolate inference-time scaling.Other work explores multi-agent cyberattack systems.Source: §5 · p. 9

6 Discussion and Conclusion

The authors report that simple methods can improve cybersecurity capability across multiple degrees of freedom with modest compute, supporting dynamic risk assessment. They note that tested modifications cover only part of the possible adversary strategies and that resource constraints matter for real-world use.

6 Discussion and Conclusion

  • The tested modifications cover only a subset of possible adversary strategies, including browsing, manual failure inspection, RL self-training, and stronger exploration methods.The authors identify these methods as directions for future work.Source: §6 · p. 9
  • The five degrees of freedom may extend beyond CTF tasks because real-world cybersecurity shares limited information access, sparse feedback, and complex reasoning.When resources are limited, iterative workflow refinement should use fast-path heuristics, real-time feedback loops, or minimal viable actions to balance accuracy and efficiency.Source: §6 · p. 9
  • 40% capability improvement is possible across multiple degrees of freedom with only 8 GPU hours or <$36 of compute.The authors argue that cybersecurity evaluations must evolve because strong verifiers and self-improvement without expert knowledge may expose less-sophisticated adversaries to these gains.Source: §6 · p. 10

NeurIPS Paper Checklist

The checklist directs readers to evidence on the paper’s claims, limitations, reproducibility, and experimental reporting, while documenting non-applicable areas and disclosures about ethics, assets, and LLM use. It also frames dynamic evaluation as support for informed cybersecurity risk decisions, acknowledges potential misuse, and records the computing setup for inference and fine-tuning.

NeurIPS Paper Checklist

  • The authors point to Section 4.5 and Figure 1b for comparative analysis supporting the paper’s main claims.Source: NeurIPS Paper Checklist · p. 16
  • The paper discusses its limitations in Section 6 and marks theory assumptions and proofs as not applicable because it includes no theory assumptions.Source: NeurIPS Paper Checklist · p. 17
  • The authors say Section B documents experiment configuration, prompts, self-training, and data curation, while Sections 4, B.7, and B.9 specify inference and training settings.Source: NeurIPS Paper Checklist · pp. 17, 19
  • The authors provide their code in the Dynamic-Risk-Assessment GitHub repository and list existing-asset licenses in Section B.2.They state that they introduce no new assets.Source: NeurIPS Paper Checklist · pp. 18, 21
  • The authors report 95% confidence intervals for all experiments and direct readers to Section B.8 for their calculation method.They also point to Sections 4.5 and B.6 for compute-cost analysis, and Sections B.1 and B.9 for compute resources.Source: NeurIPS Paper Checklist · p. 19
  • The authors state that they conform to the NeurIPS Code of Ethics, discuss societal impact in Section A, and released no new high-misuse-risk data or models.They mark crowdsourcing and human-subject research as not applicable, and say LLMs were used only for writing, editing, and formatting.Source: NeurIPS Paper Checklist · pp. 20–22

A Impact Statement

  • The authors acknowledge that their methods could increase offensive cybersecurity capability, but argue that better risk understanding and more responsible deployment outweigh the potential misuse.They warn that flawed risk assessment can create a false sense of security and affect policymaking and deployment decisions (Qi et al., 2024a).Source: A Impact Statement · p. 23

B.1 Hardware Configurations

  • Experiments used four NVIDIA H100 80GB GPUs on one node for inference and 16 H100 80GB GPUs across two nodes for Section 4.3 fine-tuning.A virtual machine with 4 AMD EPYC 7763 CPU cores and 32GB memory hosted Docker containers and queried the model on a server through vLLM (Kwon et al., 2023).Source: §B.1 · p. 23

B.2 Dataset Details

The appendix describes three evaluation datasets and the structure of an InterCode CTF challenge, including how the environment initializes and scores an agent's attempt.

B.2 Dataset Details

  • The study evaluates the agent on three datasets, with dataset links and licenses listed in Table 3.The table is captioned as covering dataset links and licenses.Source: §B.2 · p. 23, Table 3 · p. 23

B.3 Structure of the CTF Challenges

  • An InterCode CTF challenge initializes from challenge.json, may include starter files, and returns a reward based on whether the agent submits the correct flag.The built-in evaluator scores the flag submitted through the CheckFlag tool.Source: Figure 7 · p. 23, §B.3 · p. 23

B.4 Benchmark Issues

The appendix identifies benchmark issues that led to excluding erroneous InterCode tasks, then describes a stratified InterCode train-test split based on estimated task difficulty.

B.4 Benchmark Issues

  • 10 erroneous InterCode tasks were excluded after benchmark issues were identified.The listed issues were unavailable PicoCTF servers, visual flags incompatible with language-only agents, and missing essential files.Source: §B.4 · p. 24
  • PicoCTF tasks depended on unavailable servers, some challenges included visual inputs, and one challenge lacked essential files.Source: §B.4 · p. 24

B.5 Train-Test Split for InterCode CTF

  • Tasks were binned into five difficulty groups using base-agent pass@1 scores, then sampled from those bins to form a stratified train-test split.The base agent ran with N = 20 for 10 rollouts; higher pass@1 indicated an easier task.Source: §B.5 · p. 24

B.6 Details for Compute Budget Estimation

The appendix estimates compute costs for stateful and non-stateful improvement strategies and describes inference settings and bootstrap procedures used in evaluation.

B.6 Details for Compute Budget Estimation

  • Performance in stateful settings saturated at N = 40, costing 2.53 GPU Hours; self-training and its evaluation totaled 7.1 GPU Hours.Self-training included trajectory collection, five fine-tuning epochs, and test-set evaluation.Source: §B.6 · p. 24
  • Under an 8 H100 GPU Hour budget, repeated sampling allows up to 33 samples at N = 20 but only 5 at N = 50.Prompt refinement allows up to 15 iterations, while self-training or workflow refinement leaves compute for up to 3 resamples.Source: §B.6 · p. 24
  • The fixed-budget comparison was estimated at 277.01 H100 GPU Hours because each experiment was repeated at least five times for evaluation.The 8 GPU Hour cap refers to each evaluated experiment, not the aggregate cost of repeated evaluations.Source: §B.6 · p. 24, Table 4 · p. 25

B.7 Inference Parameters

  • The default inference pipeline uses N = 20, k0 = 12, temperature = 0.6, repetition_penalty = 1.0, top_p = 1.0, and max_tokens = 1024.Source: §B.7 · p. 24

B.8 Confidence Interval Computation

  • Confidence intervals are estimated by bootstrapping rollouts within each task, without resampling tasks because task-difficulty distribution is held constant.For iterative prompt refinement, the bootstrap instead samples pass@k values directly from per-k matrices.Source: §B.8 · p. 25

B.9 Training pipeline used in self-training

The appendix details training and refinement procedures, then analyzes failure patterns across repeated sampling, prompt refinement, and workflow refinement.

B.9 Training pipeline used in self-training

  • Fine-tuning converts successful multi-turn trajectories into 181 single-turn prompt-response pairs and uses the Huggingface TRL SFT Trainer.The authors report this conversion approach was more effective than training on entire trajectories; Figure 8 shows the training-data length distribution.Source: §B.9 · p. 26, Figure 8 · p. 26, Table 5 · p. 26

B.10 Prompts Used in Iterative Prompt Refinement

  • Prompt refinement adds a new generated refinement to the initial user message using the latest failed trajectory and its finish reason.Source: §B.10 · p. 26

B.11 Prompts Used in Iterative Workflow Refinement

  • Workflow refinement prompts a meta-agent to design a replacement conversation-step function, then evaluates generated workflows on InterCode CTF development tasks.The prompt asks for a Python implementation intended to maximize the fraction of CTF challenges solved.Source: §B.11 · p. 27

C Failure Mode Analysis

  • The failure analysis uses rule-based trajectory categories informed by manual inspection, including format mismatch, tunnel vision, and wrong-flag submissions.The analysis covered 40 Cybench tasks, 200 NYU CTF test tasks, and 36 InterCode CTF test tasks.Source: C Failure Mode Analysis · p. 28, Table 6 · p. 28

C.1 Repeated Sampling and Increasing Max Rounds of Interactions

  • Increasing interaction rounds reduced max-round failures but was accompanied by a shift toward format mismatches.The analysis covered repeated sampling at k = 10 and N = 10, 20, and 30 across three benchmarks.Source: §C.1 · p. 28, Table 7 · p. 28, Figure 9 · p. 29

C.2 Iterative Prompt Refinement

  • The prompt-refinement analysis compares iterative rounds with repeated sampling and uses bootstrapping to estimate aggregated failure distributions across runs.For each bootstrap sample, it identifies tasks that fail in all sampled runs and repeats the process 5,000 times.Source: §C.2 · p. 28, Figure 9 · p. 29, Figure 10 · p. 29
  • Iterative prompt refinement reduced average failed tasks by 58.6%, from 16.20 to 6.70, versus a 53.64% reduction, from 15.08 to 7.29, for repeated sampling.Format mismatches fell from 2.90 to 0.20, max-round failures from 7.3 to 3.1, and tunnel-vision cases from 0.9 to 0.6.Source: §C.2 · p. 29, Figure 10 · p. 29, Table 8 · p. 29

C.3 Self-Training

Self-training improves format compliance while increasing tunnel vision; iterative workflow refinement mitigates several observed failure modes.

C.3 Self-Training

  • Across the self-training experiment evaluating 0, 5, and 10 epochs, format mismatches fell 17%, from 2.9 to 2.4, indicating improved adherence to scaffolding requirements.The experiment also found doubled tunnel-vision instances; the authors suggest self-training may limit exploration diversity.Source: §C.3 · p. 30

C.4 Iterative Workflow Refinement

  • Workflow refinement at iterations 2 and 9 mitigated format mismatches, context-window failures, tunnel vision, and wrong-flag errors.The authors use these results to support the importance of workflow design and scaffolding for agent performance.Source: §C.4 · pp. 30–31, Table 11 · p. 31

D.1 Qualitative Examples on Increasing Max Rounds of Interactions

The max-round example shows that adding interaction budget can leave performance constrained by repetition, while iterative prompt refinement can guide a successful solve.

D.1 Qualitative Examples on Increasing Max Rounds of Interactions

  • With N = 60, the agent repeats the same content from round 5 until the limit, so increasing rounds alone offers limited benefit.The repeated trajectory can constrain performance gains.Source: §D.1 · p. 32

D.2 Qualitative Examples on Iterative Prompt Refinement

  • Following its generated instructions, the agent eventually solves the cryptography challenge and submits a correct flag.The example begins with an encrypted file that provides no direct information about the encryption method.Source: §D.2 · pp. 32–33
  • The prompt-refinement strategy redirects the agent from brute force toward trying cryptographic terms, phrases, and key variations.The generated instructions include appending underscores, numbers, and special characters.Source: §D.2 · p. 33

E Additional Results for Gemma3-12B-It Model

On InterCode CTF, increasing repeated sampling and interaction rounds improves performance, while iterative prompt refinement is more efficient than repeated sampling at the same rollout count. Self-training and workflow refinement are harder for weaker models.

E Additional Results for Gemma3-12B-It Model

  • For self-training, the study fine-tunes on converted training trajectories; for workflow refinement, it prompts the model 20 times to generate refinements.The workflow-refinement attempt failed to follow instructions and provide reasonable refinements.Source: E Additional Results for Gemma3-12B-It… · p. 34
  • Increasing k and N consistently improves performance, and iterative prompt refinement is more efficient than repeated sampling under the same number of rollouts.Figure 12 reports Gemma3-12B-It performance on InterCode CTF (Test) under NYU CTF agent scaffolding.Source: Figure 12 · p. 34, E Additional Results for Gemma3-12B-It… · p. 34
  • Self-training and iterative workflow refinement yield harder-to-obtain gains for weaker models, which struggle to generate successful trajectories or follow long-context rubrics.The self-training experiment used 25 InterCode CTF training trajectories converted into 126 single-turn conversations; workflow refinement failed to produce reasonable refinements.Source: E Additional Results for Gemma3-12B-It… · p. 34
Loading 2505.18384v5…