Source-linked AI summary
Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios
Linus Folkerts, Will Payne, Simon Inman, Philippos Giavridis, Joe Skinner, Sam Deverett, James Aung, Ekin Zorer, Michael Schmatz, Mahmoud Ghanem, John Wilkinson, Alan Steer, Vy Hong, Jessica Wang
TL;DR
The paper asks whether frontier AI models can autonomously execute extended cyberattack chains, a capability that existing isolated-task evaluations do not adequately measure. It evaluates seven models on two multi-step cyber ranges across model generations and inference-time compute budgets. Performance improves with both more compute and newer models on the corporate range, while industrial-control performance remains limited.
Problem
Existing cyber evaluations often use isolated CTF or question-answer tasks that do not capture autonomous multi-step execution, state tracking, and error recovery in large-scale environments.
Method
The study compares seven models across two cyber ranges—a 32-step corporate attack and a 7-step industrial-control attack—at varying inference-time compute budgets.
Results
Performance consistently improves with increased inference-time compute and newer model releases; at 10M tokens, corporate-range averages rise from 1.7 steps for GPT-4o to 9.8 for Opus 4.6, while the best run completes 22 of 32 steps.
Takeaways & Limitations
Autonomous cyber capability is advancing on corporate network attacks, with gains from both additional tokens and successive model generations, but industrial-control performance remains substantially lower.
Takeaways & Limitations
The results measure capability without active defensive responses, because alerts are recorded but do not impede, slow, or trigger defenses.
Abstract
from arXiv · showhide
We evaluate the autonomous cyber-attack capabilities of frontier AI models on two purpose-built cyber ranges-a 32-step corporate network attack and a 7-step industrial control system attack-that require chaining heterogeneous capabilities across extended action sequences. By comparing seven models released over an eighteen-month period (August 2024 to February 2026) at varying inference-time compute budgets, we observe two capability trends. First, model performance scales log-linearly with inference-time compute, with no observed plateau-increasing from 10M to 100M tokens yields gains of up to 59%, requiring no specific technical sophistication from the operator. Second, each successive model generation outperforms its predecessor at fixed token budgets: on the corporate network range, average steps completed at 10M tokens rose from 1.7 (GPT-4o, August 2024) to 9.8 (Opus 4.6, February 2026). The best single run completed 22 of 32 steps, corresponding to roughly 6 of the estimated 14 hours a human expert would need. On the industrial control system range, performance remains limited, though the most recent models are the first to reliably complete steps, averaging 1.2-1.4 of 7 (max 3).
1. Introduction
Whether frontier AI models can execute extended cyberattack chains in complex environments, and how quickly this capability is improving, remains unclear. The paper addresses this gap by evaluating models on multi-step corporate and industrial-control attack ranges rather than isolated challenges.
- Autonomous execution of extended attack chains remains an unresolved capability question for frontier AI models.
- Existing CTF and question-answer evaluations miss the autonomy, multi-step reasoning, state tracking, and error recovery required in large-scale network environments.
- The evaluation measures the number of attack steps completed autonomously and compares performance across inference-time compute and model generations.
2. Results
Across the two cyber ranges, autonomous attack performance improves with more inference-time compute and newer model releases, but progress remains uneven across attack phases and tasks. Corporate-network performance reaches substantially farther than industrial-control performance, while individual runs show high variance and some unintended attack paths.
- Scaling inference-time compute: Performance scales logarithmically with total tokens, with no observed plateau up to 100M tokens.Most attempts were still advancing or recovering from failed approaches when the token limit was reached.
- Scaling inference-time compute: Scaling inference-time compute requires no specific technical sophistication, making larger token budgets an accessible route to improved performance.The paper contrasts this with custom scaffolding, expert prompting, and tailored tooling.
- Scaling inference-time compute: Individual Opus 4.6 runs varied from 11 to 22 of 32 completed steps, leaving the best fixed-budget allocation strategy unresolved.The paper identifies a choice between fewer long runs with context compaction and many shorter independent attempts.
- Performance improves with each model release: GPT-4o averaged 1.7 steps at 10M tokens, versus 9.8 steps for Opus 4.6 at the same budget.The intervening averages were 5.8 for Sonnet 3.7, 7.6 for Opus 4.5, and 8.0 for 5.1 Codex.
- Performance improves with each model release: At 100M tokens, Opus 4.5 averaged 11.0 steps and Opus 4.6 averaged 15.6, a 42% improvement; Opus 4.6’s best run reached 22 of 32 steps.The best Opus 4.6 run corresponded to roughly 6 of the 14 hours estimated for a human expert.
- Attack-chain bottlenecks: The corporate attack has bottlenecks at milestones 4–5 and 7–8, while Opus 4.6 first reliably overcame the milestone-4 NTLM relay bottleneck.The later bottleneck involves chained CI/CD pipeline attacks requiring credential discovery, configuration manipulation, and backdoored artifact deployment.
- Performance on Cooling Tower: On Cooling Tower, Opus 4.6 averaged 1.4 of 7 steps at 100M tokens, while GPT 5.3 Codex reached a maximum of 3 steps.The range’s steps represent larger work units with more complex dependencies and more opportunities for distraction.
- Performance on Cooling Tower: Several models bypassed the intended Cooling Tower path by probing the PLC protocol directly, and some exploited an unintended authentication bug that has since been patched.Direct protocol probing obtained Step 4; brute-forcing session identifiers could obtain Step 6 without the intended cryptographic material.
3. Methodology
The study measures autonomous cyber capability on two multi-step cyber ranges using standardized agents, multiple frontier models, and varying token budgets. Performance is counted through verified attack-step completion, while the ranges and metric remain simplified relative to real-world operations.
- Evaluation environments: Cyber ranges simulate corporate and industrial environments with vulnerabilities arranged into sequential attack chains and defined objectives.“The Last Ones” requires corporate data exfiltration across 32 steps, while “Cooling Tower” requires disrupting a simulated power plant across 7 steps.
- Metrics: Step completion is verified by submitted flags with binary credit, using furthest-chain progress for “The Last Ones” and directly observed steps for “Cooling Tower.”The two ranges use different counting rules because the corporate attack is largely sequential whereas the industrial-control steps are more independent.
- Models and budgets: The evaluation compares seven models released from August 2024 to February 2026 at 10M- and 100M-token budgets.Recent models were run end-to-end on both ranges at 100M tokens, while all seven models received additional 10M-token evaluations on starting points.
- Agent design: Agents use a minimal ReAct-style loop in Kali Linux, cycling through reasoning, command execution, and observation.The setup permits Bash, Python, and Mythic commands and is designed to isolate the underlying model rather than specialized scaffolding.
- Scope and assumptions: The ranges omit active defense and several real-world attack phases, so completed-step counts do not represent full autonomous attack capability.The metric excludes target selection, initial-access decisions, persistence under incident response, and attack-infrastructure setup; alerts are recorded but do not impede agents.
4. Related work
Prior cyber evaluations emphasize isolated challenges, language tasks, or individual offensive operations, while broader agent benchmarks measure long-horizon execution in realistic environments. This work positions its cyber ranges as an offensive-security analogue combining those evaluation goals.
- Cyber benchmarks: CTF and cybersecurity-NLP benchmarks measure specific skills but generally do not capture extended autonomous attack execution.Examples include NYU CTF Bench, InterCode-CTF, Cybench, and CyberBench.
- Agentic benchmarks beyond cyber: SWE-bench, WebArena, and GAIA evaluate agents on extended multi-step tasks involving realistic environments, browsing, reasoning, or tool use.These benchmarks provide non-cyber precedents for measuring long-horizon autonomous execution.
- Positioning: The cyber ranges provide the offensive-security analogue of broader agentic benchmarks by requiring long-horizon execution in adversarial network environments.Unlike many isolated tasks, the agent must discover task structure through reconnaissance.
- Offensive cyber operations: Prior offensive-cyber studies examine multi-step exploitation, hierarchical agent teams, and comparisons between AI agents and human professionals.The cited work includes exploitation of web vulnerabilities, zero-day exploitation by agent teams, and live-enterprise comparisons.
- Frontier evaluation: Frontier-AI evaluation frameworks motivate measuring autonomous capability through task duration and time horizons.The paper situates its cyber evaluation within work on dangerous capabilities, autonomous replication, and adaptation.
5. Limitations
The results are bounded by simplified environments, incomplete operational-security measurement, and experimental choices that may understate or constrain observed capability. They should therefore be interpreted as evidence from synthetic ranges rather than complete real-world attack performance.
- Environment realism: The ranges lack active defenders, so security teams cannot monitor intrusions, respond to alerts, or adapt defenses during attacks.Elastic Defend was not configured to block or impede attack progress.
- Operational security: Triggered detections are recorded but not penalized, so higher task completion may coexist with poorer operational security.A model completing more steps while triggering many alerts may represent a lesser threat than a model progressing reliably while remaining undetected.
- Environment realism: The ranges contain deliberately high vulnerability density and fewer nodes, services, and files than typical production networks.They are more complex than CTFs but remain considerably simpler than real enterprise environments.
- Experimental design: The experiments may understate capability because they use 10M- and 100M-token budgets, minimal scaffolding, and untailored tooling.The authors state that higher budgets, custom frameworks, and model-specific tool optimization would likely improve performance.
- Threat model: The fully autonomous-agent setting omits human-AI teaming, although targeted human intervention could address specialist-knowledge, tracking, and long-sequence failures.The authors identify human operators using agents at specific bottlenecks as a potentially more operationally relevant threat model.
- Benchmark longevity: Publishing range descriptions and results may reduce benchmark longevity by making future evaluations less held out.The authors prioritize transparency as a longitudinal snapshot of capability.
6. Conclusion
The paper finds reinforcing gains from more inference-time compute and newer model releases on a corporate attack range, while industrial-control performance remains limited. It presents cyber ranges as a longitudinal measurement instrument but emphasizes synthetic scope, domain coverage, and defensive realism as priorities for interpretation and future work.
- Conclusion: Seven models evaluated across two multi-step ranges provide a longitudinal view of autonomous cyber capability from August 2024 to February 2026.The ranges cover a 32-step corporate attack and a 7-step industrial-control attack with heterogeneous attack chains.
- Conclusion: At 10M tokens, average corporate-range progress rose from 1.7 steps for GPT-4o to 9.8 for Opus 4.6, while 10M-to-100M-token increases yielded gains of up to 59%.The best single run completed 22 of 32 steps, corresponding to roughly 6 of an estimated 14 human hours.
- Conclusion: Both increased inference-time compute and newer model releases consistently improve performance, although industrial-control performance remains limited.This is the paper’s central cross-task capability trend.
- Future work: VM-based ranges provide realism but impose substantial engineering and operational overhead, motivating higher-fidelity containerized and digital-twin alternatives.Such approaches could expand evaluation scope and scale, including for industrial-control infrastructure.
- Future work: The two ranges reveal different capability levels but are insufficient for general conclusions across domains.Broader coverage of domains, vulnerability types, and vulnerability densities could distinguish general from domain-specific bottlenecks.
- Future work: Real-world penetration-testing engagements could test whether synthetic-range capabilities translate to operational risk and expose failure modes absent from these evaluations.The authors state that current capability has reached a level where such testing is feasible.
Impact Statement
The paper evaluates autonomous cyberattack capabilities while acknowledging that detailed knowledge of AI-driven attacks could be misused. Its cyber ranges are simulated environments and do not target real systems.
- The paper evaluates frontier AI models’ autonomous cyberattack capabilities to inform safeguards and policies.
- The authors limit methodological detail to support reproducibility and scientific scrutiny without providing a step-by-step guide.
- The cyber ranges are simulated environments and do not target real systems.
A. Step-level performance of different models on “The Last Ones”
Figure 3 compares per-step completion rates across all models on the 32-step corporate attack at a 100M-token budget, with five runs per model.
- Each model’s per-step rates are based on 5 runs.
- Human expert time estimates are shown above the milestones for comparison.
B. Model performance from different starting points in the attack chain
Performance is stronger on early reconnaissance and web-exploitation milestones, then drops on later specialist steps; mid-range starts expose long-horizon difficulty beyond isolated step capability.
- A. Step-level performance of different models on “The Last Ones”: Up to step 13, models perform comparatively well on milestones 1–4, which primarily involve reconnaissance and web exploitation.
- A. Step-level performance of different models on “The Last Ones”: Performance drops sharply on milestones 5–9, which require reverse engineering, cryptography, and malware development.
- B. Model performance from different starting points in the attack chain: Step success rates vary substantially across the attack chain because context accumulation and inherent step difficulty affect later attempts.
- B. Model performance from different starting points in the attack chain: Later steps require 8–16 actions and specialist expertise, compared with 2–4 actions for milestones 1–2.
- B. Model performance from different starting points in the attack chain: The most capable models struggle most with milestones 5, 7, 8, and 9 because of long sequences, specialist knowledge, and heavy information dependencies.
- B. Model performance from different starting points in the attack chain: Mid-range starts enable completion of steps missed in end-to-end runs, while performance still drops on later steps regardless of starting point.
C. Attack chain details
The paper defines two multi-stage attack chains: a 32-step corporate network scenario and a 7-step industrial control system scenario, each spanning heterogeneous technical phases.
- “The Last Ones” groups 32 steps into 9 milestones spanning reconnaissance, lateral movement, credential theft, reverse engineering, C2 exploitation, supply chain compromise, and data exfiltration.
- “Cooling Tower” contains 7 steps from HMI web exploitation through proprietary protocol and cryptographic reverse engineering to PLC register manipulation.
D. CTF performance context
The paper contextualizes cyber-range results with broader isolated CTF evaluations, finding that improving CTF performance reflects broader cyber-skill gains but does not reliably predict multi-step autonomous execution.
- CTF evaluations test specific cyber skills such as web exploitation, cryptography, and reverse engineering.
- Harder CTF performance does not necessarily translate into effective skill chaining in multi-step scenarios.
- The gap between isolated skill and autonomous execution is a key finding of the paper.
- 71 AISI cyber evaluations were each repeated 10 times, with performance measured as average success across the 10 tries.
E. Full per-run results
The full per-run results record step-level outcomes across models, token budgets, starting points, and epochs, while distinguishing independent flags, partial starts, and completed prerequisites.
- Each row represents one attempt grouped by model, token budget, starting point, and epoch.
- X marks a found flag, a dash marks an unsuccessful flag, and blank cells mark steps completed before the run began.
- Not all flags within a milestone are required for its overall step count to advance because some steps can be completed independently.
- Tables 2 and 3 provide complete per-run subflag and flag outcomes for the TLO and Cooling Tower evaluations.
- Partial starting points begin after an intermediate reconnaissance milestone but before its corresponding flag is captured.