Source-linked AI summary
Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents
Saeedeh Lohrasbi, Mohammad Mamun, Ahmed Yehia, Scott Buffett, Sherif Saad
TL;DR
Final success can conceal costly repetition and misinterpretation of execution evidence, leaving the location and persistent causes of cyber-agent bottlenecks as complementary diagnostic questions. The study combines subtask resource localization with trajectory diagnosis across selected models, scenarios, and scaffolding modes. Across complete judge panels, diagnosed bottlenecks varied by scenario and scaffolding mode rather than following a uniform trend with autonomy.
Problem
Interactive benchmarks assess execution-grounded capability, but where progress becomes constrained within observed cyber-agent executions and which persistent weakness explains it remain complementary questions.
Source: §1 · p. 1Method
The study combines subtask-level resource localization with trajectory-level diagnosis across selected high-performing trajectories, two scenarios, six acting models, and three scaffolding modes.
Source: §1 · p. 2Results
Across 29 complete judge panels, bottlenecks varied by scenario and scaffolding mode across planning, execution, recovery, and framework effects rather than following a uniform trend with autonomy.
Source: §6 · p. 11Takeaways & Limitations
Evaluation should assess verified progress, unresolved prerequisites, evidence interpretation, and whether retries reflect meaningful adaptation, not just final success.
Source: §6 · p. 12Findings are diagnostic evidence from selected high-performing trajectories in two scenarios, and judge agreement does not substitute for independent human validation.
Source: §6 · p. 12
Abstract
from arXiv · showhide
Multi-stage LLM-based cyber agents may complete attack workflows while remaining brittle, costly, or reliant on incorrect interpretations of execution evidence. Success rates alone obscure inefficiency, adaptation through retries, and recognition of success or failure. We present an end-to-end diagnostic study of an Autonomous Adversary system with orchestrator, executor, and validator LLMs in enterprise-like lateral-movement scenarios. Six frontier models are evaluated across two scenarios and three modes: expert-defined, self-scaffolded, and fully autonomous. We assess validator consistency and evidence grounding; introduce a subtask-conditioned, cost-aware score for abnormal token use, retries, and runtime; and use comparative LLM-as-a-Judge analysis to identify planning deficiencies, including tool misalignment, plan similarity, over-specification, inadequate probing, and weak recovery. Validators are generally relevant and evidence-grounded but often nonspecific and overly optimistic. Bottlenecks cluster in credential and lateral-movement tasks, spread with scenario complexity, and vary more under full autonomy. Reliable evaluation must assess outcomes, evidence interpretation, resource use, and adaptation after failure.
1 Introduction
Cyber-agent workflows can reach success while incurring costly repetition or misreading execution evidence, so final success alone misses important distinctions. This study diagnoses where progress is constrained and which persistent weaknesses explain it, separating resource use from behavior and agent actions from orchestration and validation.
1 Introduction
- Cybersecurity workflows depend on credential acquisition, identity transition, lateral movement, and cleanup, but command execution may not establish the state needed for the next stage.Eventual success can conceal costly repetition, and a final success indicator captures neither that nor misinterpretation of execution evidence.Source: §1 · p. 1
- The study asks where progress becomes constrained within observed cyber-agent executions and what persistent weakness best explains each constraint.This diagnostic question complements interactive benchmarks that evaluate execution-grounded capabilities.Source: §1 · p. 1
- The analysis separates resource consumption from behavioral interpretation and distinguishes the acting model from orchestration and validation components.The comparison unit varies with task structure: shared expert decompositions enable aligned subtask comparisons, while agent-generated objectives require trajectory-level diagnosis.Source: §1 · p. 2
- The contributions include a two-resolution diagnostic framework, separation of bottlenecks from mechanisms and recovery, and a case study of scenario-dependent scaffolding relations.The work is diagnostic, not a representative model ranking or an experimental demonstration of causality.Source: §1 · p. 2
2 Related Work
Prior work evaluates cyber capabilities, tool-using failures, and LLM-judge assessment, while this study focuses on diagnosing persistent difficulty within observed cyber-agent executions.
2 Related Work
- Interactive benchmarks such as InterCode and CyBench evaluate execution-grounded capabilities, whereas this analysis diagnoses persistent difficulty within two enterprise-like lateral-movement workflows.A capability score and a diagnosis of persistent difficulty answer different questions.Source: §2 · p. 2
- Failure taxonomies distinguish reasoning, tool-interaction, and system faults; this study combines categorical diagnosis with resource signals and separates isolated errors from persistent weaknesses.Task-level labels and trajectory-level labels serve different purposes, since local plan properties need not match the dominant explanation for a full run.Source: §2 · p. 2
- Multiple LLM judges expose evaluator disagreement, with consensus treated as consistency rather than human-validated correctness.The figure identifies the Autonomous Adversary execution architecture and evidence flow used in the analysis.Source: §2 · p. 2, Figure 1 · p. 3
3 Experimental System and Study Design
The study analyzes a controlled Autonomous Adversary environment across distinct scaffolding modes, scenarios, models, and selected trajectories. The setup supports diagnosis of execution bottlenecks but limits claims about general prevalence and model variability.
3.1 Execution Architecture and Scaffolding
- The controlled environment includes segmented subnets, remote-administration services, domain infrastructure, and security misconfigurations, but no active defender or execution-blocking detection penalty.Findings concern planning, execution, state interpretation, and recovery in this setting, not resilience against adaptive defensive intervention.Source: §3.1 · p. 3
- The orchestrator proposes plans, the executor returns command and tool evidence, and the validator decides whether to continue, retry, submit, or halt.A validator-accepted outcome is not necessarily independently verified; the framework also has memory and no fixed tool list, with agents generating abilities through the executor.Source: §3.1 · pp. 3–4
- Expert-defined mode supplies ordered subtasks and verification criteria, self-scaffolded mode lets the model generate them, and fully autonomous mode adds reconnaissance and objective ordering.The expert-defined decomposition was developed by an offensive cybersecurity expert who independently planned and solved the scenarios.Source: §3.1 · p. 4
3.2 Scenarios, Budgets, and Selection
- S1 begins at a low-privilege workstation with an active administrator session, while S2 begins with an administrative jump-server foothold and has different dependencies and starting privileges.The scenarios share an environment range but differ in their task structure.Source: §3.2 · p. 4
- Six acting models were evaluated, with four models selected for cross-setting comparisons because they had the most consistent trajectory coverage.The Claude Opus 4.5 S2-A panel was incomplete and excluded from corresponding aggregate comparisons; Table 1 presents primary diagnoses as modal votes, not success rates.Source: §3.2 · p. 4, Table 1 · p. 4
- A 45,000-token reference applies to one LLM call rather than an entire run, and the study does not experimentally isolate a budget effect.For each available model–scenario–mode configuration, analysis selected the run with greatest judge-verified task progress, so results represent best-case observed performance rather than prevalence or average performance.Source: §3.2 · p. 5
4 Multi-Resolution Diagnostic Method
The diagnostic method pairs subtask-level resource comparisons with trajectory-level judgments of persistent bottlenecks, using complete execution traces and multiple evaluators. Its outputs distinguish relative resource amplification, local plan properties, run-level diagnoses, recovery, and evaluator support.
4.1 Task-Level Localization
- For aligned expert-defined subtasks, the method measures tokens, elapsed time, and attempts, then scores positive standardized resource deviations with equal weights.The score is cohort-dependent, not an absolute difficulty scale; zero does not establish success or intrinsic ease, and the measure is not used across agent-generated subtask indices.Source: §4.1 · p. 5
- A planning judge labels observable plans for similarity, dependencies, over-specification, missing environmental probing, recovery paths, tool alignment, and inefficient depth.These labels describe visible plan structure, not hidden reasoning, and the single planning judge provides interpretation rather than validated causal attribution.Source: §4.1 · pp. 5–6
4.2 Trajectory Taxonomy and Evidence
- Trajectory judges review traces that preserve plans, actions, evidence, validator decisions, retries, revisions, state updates, transitions, and outcomes.A bottleneck is a persistent constraint on progress or avoidable resource use; judges distinguish symptom, immediate cause, and hypothesized root cause, which is diagnostic rather than intervention-established.Source: §4.2 · p. 6
- Meaningful adaptation changes a hypothesis, resolves a prerequisite, gathers discriminating evidence, changes dependencies, or adopts a substantively different strategy.Changing syntax or tools while preserving a failed assumption counts as repetition; validator, framework, or environment contributions can coexist with an agent-side bottleneck.Source: §4.2 · pp. 6–7
4.3 Evaluator Roles and Aggregation
- Trajectory judges assess complete logs, while the online validator affects execution and other evaluation components assess recorded evidence offline.The system validator uses the acting model; judge outputs include structured progress, categories, tags, severity, confidence, and evidence-based explanations, with invalid or failed calls left missing.Source: Table 3 · p. 7, §4.3 · pp. 7–8
- Modal diagnosis support counts valid judges, preserves ties, and does not treat a unique mode as necessarily a majority.Mechanism, adaptation, and contribution counts are descriptive outputs nested within trajectories, not independent agent trials.Source: §4.3 · p. 8
5 Results
Across the evaluated runs, bottlenecks varied by scenario and scaffolding mode, while validator explanations were often evidence-grounded but less specific. The results combine resource localization with trajectory diagnoses, adaptation judgments, and analysis of recurring mechanisms.
5.1 Validator Evidence Assessment
- Validator-explanation quality scored 0.79 for successful attempts and 0.84 for unsuccessful attempts; grounding scores were 0.81 and 0.90, while specificity was 0.69 and 0.68.Cautiousness was 0.70 for successful versus 0.92 for unsuccessful verdicts; these judge scores assess explanations, not success-classification accuracy or validator calibration, and lacked human reference labels.Source: §5.1 · p. 8
5.2 Localized Resource Amplification
- Figure 2 shows isolated high-score cells in S1 and repeated elevations in one plotted S2 run; comparable resource scores received different planning diagnoses, making localization and interpretation complementary.Scores indicate within-subtask relative resource amplification; blank cells mean the agent never reached those subtasks, not zero resource use, and the score–severity relationship is not statistically validated.Source: §5.2 · p. 8, Figure 2 · p. 9
5.3 Primary Trajectory Diagnoses
- Across 23 completed comparison-cohort trajectories, S1-E diagnoses were action-execution modes for Opus, Gemini, and GPT-5.1 (Sonnet tied), while all S1-S trajectories were planning-limited (4/5–5/5).The S1-E action-execution mode had 2/5 support for Gemini and 5/5 for GPT-5.1; model-generated decomposition was associated with difficulty constructing and validating dependencies, not a controlled causal effect.Source: §5.3 · p. 9
- In S2, expert-mode diagnoses varied by model, self-scaffolded trajectories were mostly action-execution-limited, and all three completed S2-A panels were planning-limited with 3/5 support; Opus’s panel was incomplete.Recurring self-scaffolded blockages involved agent deployment with wrong executables, unresolved variables, placeholder commands, or treating process creation as an operational callback.Source: §5.3 · p. 10
5.4 Agreement, Adaptation, and Mechanisms
- Agreement was lower in S2, where prerequisite, command, and validation problems can coexist; mechanism counts emphasize command construction in S1-E and unsupported assumptions and missing prerequisites in S1-S.In the comparison cohort, 12 of 23 panels had at least 4/5 support, guaranteeing a unique mode after any one-judge deletion; validator/framework contributions appeared in 72–100% of included judgments, with severity means from 4.03 to 5.00.Source: §5.4 · pp. 10–11
- No-meaningful-adaptation judgments in the S1 comparison cohort rose from 5/20 under expert decomposition to 15/20 under self-scaffolding and 20/20 under full autonomy; both S2 settings had 20/20.The selected S1-S Opus trace was a counterexample: it switched from unsuccessful credential extraction to password spraying and completed the objective; these selected traces do not show autonomy generally reduces recovery.Source: §5.4 · pp. 10–11
6 Discussion and Conclusion
The framework combines subtask-level resource localization with trajectory-level diagnosis, and selected traces show bottlenecks vary by scenario and scaffolding mode. The findings motivate evaluating progress, evidence interpretation, resource use, and adaptation beyond final success.
Discussion and Conclusion
- Across 29 complete judge panels, observed bottlenecks varied by scenario and scaffolding mode, spanning planning, execution, recovery, and framework effects.The pattern did not follow a uniform trend with autonomy.Source: §6 · p. 11
- Evaluation beyond final success should include verified progress, unresolved prerequisites, evidence interpretation, and whether retries reflect meaningful adaptation.Source: §6 · p. 12
- The findings are diagnostic evidence from selected high-performing trajectories, not estimates of model-level failure prevalence.The study covers two lateral-movement scenarios in one environment; incomplete coverage and best-trajectory selection limit conclusions about within-model variability, and judge agreement does not substitute for independent human validation.Source: §6 · p. 12