PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Making

Pasquale Malacaria, Yunxiao Zhang

arXiv:2608.20966v1cs.CRcs.AI

TL;DR

The paper asks whether LLMs can perform structured cybersecurity defence selection over attack graphs rather than rely on familiar controls, domain priors, or framing. It compares LLM-generated portfolios and solvers with a game-theoretic optimization reference across controlled threat scenarios. LLMs show conditional competence, often aligning with the formal objective, but their behaviour is not robust and generated solvers scale poorly.

  • Problem

    It is unclear whether apparently strong LLM cybersecurity decisions reflect reasoning over attack topology or familiar controls, domain priors, and evaluation framing.

  • Method

    The paper evaluates LLM-selected defence portfolios and generated solvers on attack-graph tasks, comparing portfolios with a game-theoretic optimization reference.

  • Results

    LLMs often produce strong strategies aligned with the optimization objective, but small representation or framing changes can cause significant deviations, and generated solvers struggle to scale.

  • Takeaways & Limitations

    LLMs are conditionally competent rather than reliably optimal for structured cybersecurity decision-making.

  • Takeaways & Limitations

    Alignment with the optimization baseline decreases as graph complexity increases, while the study deliberately separates qualitative LLM reasoning from undisclosed quantitative risk mapping.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in cybersecurity workflows, yet it remains unclear whether they can perform structured security reasoning or merely rely on superficial cues and prior knowledge. We study this question in the context of defence selection over attack graphs derived from real-world threat scenarios, including ransomware, supply-chain compromise, cloud abuse, Kubernetes attacks, POS malware, and ICS/OT intrusion. Given a budget constraint, LLMs must select security controls to minimise attacker success. We compare their strategies against each other and against a game-theoretic optimization baseline used as a normative reference for structured reasoning. Our results show that LLMs exhibit conditional competence. When explicit attack-graph structure is provided, they often produce coherent strategies close to the optimization baseline. However, their capabilities are fragile. LLM behaviour becomes increasingly fragile with graph complexity and is highly sensitive to framing. Small prompt changes can substantially alter rankings, and merely relabeling a poor strategy as ``optimal'' dramatically improves its evaluation. We further observe a non-monotonic relationship between formal risk and LLM judgement: strategies closest to the optimum are not necessarily ranked highest by LLM evaluators. To further probe reasoning ability, we ask LLMs to generate solvers for the same optimization problem. While the generated implementations recover the correct high-level formulation, they scale poorly compared to a purpose-built solver. Overall, our findings show that LLMs can approximate structured cybersecurity reasoning under controlled representations, but do not apply it robustly. This has important implications for the design and evaluation of AI-assisted security decision-support systems.

1 Introduction

The paper evaluates whether frontier LLMs can make structured cybersecurity defence decisions over attack graphs, finding useful but fragile competence. Their strategies can be strong, yet evaluations shift with framing and naming, while generated solvers struggle to scale.

  • Study design: The study tests frontier LLMs as cybersecurity decision-makers selecting defence portfolios across seven real-world attack-graph scenarios and budget levels.Scenarios include ransomware, supply-chain compromise, cloud abuse, POS attacks, Kubernetes compromise, and ICS/OT intrusion.
  • Study design: A game-theoretic solution provides an independent normative reference for assessing whether LLM strategies reflect structured attacker–defender reasoning.
  • Findings: LLMs can produce strong defence strategies and discriminate between good and poor ones, but their evaluations are sensitive to framing and naming.A poor strategy labeled “optimal” may outrank genuinely strong strategies.
  • Solver generation: Generated defence solvers recover optimal solutions but struggle to scale to large attack graphs.
  • Contributions: The evaluation framework systematically examines when structured decision-making succeeds, fails, and how those failures arise.

2 Related Work

Prior cybersecurity decision-support work includes qualitative judgement and quantitative adversarial models, while LLM research has emphasized knowledge-intensive security tasks. This paper addresses the comparatively understudied question of whether frontier LLMs reason consistently over explicit attack graphs and budget-constrained control portfolios.

  • Cybersecurity decision-making: Cybersecurity investment requires selecting assets and controls while maximizing protection and limiting cost and operational disruption.
  • Existing approaches: Qualitative methods use expert judgement and relative scales, whereas quantitative methods estimate likelihood, loss, cost, and risk reduction numerically.
  • Existing approaches: Stackelberg security games model defenders committing strategies first while anticipating attackers’ best responses, often over attack graphs.
  • LLMs in cybersecurity: Prior LLM cybersecurity research has examined domain knowledge, threat patterns, recommendations, incident response, training, and structured threat-intelligence analysis.
  • Research gap: Comparatively little work tests whether frontier LLMs consistently select budget-constrained control portfolios over explicit attack graphs.It remains unclear whether strong decisions reflect attack-topology reasoning or familiar controls, domain priors, and evaluation framing.
  • This paper: The paper compares LLM portfolios with formal game-theoretic optimization and applies controlled perturbations to framing, naming, semantic information, graph structure, and complexity.

3 Cybersecurity Decision Making

The paper formulates cybersecurity defence selection as a budget-constrained Stackelberg problem over attack graphs, minimizing worst-case attacker success. It uses game-theoretic optimization as a controlled normative reference, while noting that numerical control-effectiveness estimates limit real-world validation.

  • Problem formulation: The defender selects a budget-feasible subset of controls from an attack graph to minimise a risk functional.The graph specifies nodes, edges, controls, costs, indirect costs, effectiveness, and edge applicability.
  • Problem formulation: Risk is the maximum attacker success probability over all source-to-target paths after the defender commits to a strategy.Edge success probabilities reflect the multiplicative effects of selected applicable controls.
  • Model qualification: The optimization’s “optimal” strategy is optimal only under the specified attack-graph model and assumed parameters, not empirically validated real-world ground truth.The exact effectiveness parameters are not empirically observable, and the illustrative values are included for understanding.
  • Illustrative example: In the illustrative graph, a two-control budget yields s*= [c1, c3] as the optimal portfolio under the stated parameters.The example assigns control effectiveness values of 0.4, 0.5, 0.1, and 0.1 to c1 through c4.
  • Efficient optimization: A logarithmic reformulation converts path-probability products into sums and enables a mixed-integer linear-programming formulation solvable by standard optimizers.The resulting solver handles large graphs efficiently; game-theoretic solutions in this paper take seconds, whereas LLM-generated solutions typically take minutes per instance.

4 Methodology

The methodology evaluates defence selection across realistic attack-graph scenarios constructed from documented threats. Graphs remain interpretable for LLM evaluation, while computational scalability is assessed separately on larger automatically generated graphs.

  • Threat scenarios: The study considers seven realistic threat scenarios, including ICS/OT intrusion, ransomware, supply-chain compromise, cloud abuse, POS attacks, and Kubernetes compromise.The scenarios vary in graph size, number of paths, and available controls.
  • Threat scenarios: The scenarios are extrapolated from documented real-world incidents or official kill-chain analyses and constructed for probabilistic modeling.The graphs aim to represent documented attack paths faithfully while remaining suitable for analysis.
  • Graph representation: Each graph contains nodes and edges, available controls, control costs and effectiveness, and the controls applicable to each edge.This representation links defensive choices to specific attack steps.
  • Experimental scope: The chosen graph sizes prioritize interpretable, manually inspectable threat models rather than optimization scalability.Scalability is evaluated separately using automatically generated attack graphs of increasing size.
  • Ransomware scenario: The ransomware graph models stages from initial access through execution, credential access, lateral movement, data staging, and impact.Its main path includes exfiltration and encryption, while a dashed edge represents direct encryption without exfiltration.
  • Defensive controls: Controls reduce edge-traversal success probabilities by preventing steps, limiting capabilities, or detecting and responding to activity.Examples include MFA, segmentation, EDR, SIEM, and incident response.

5 Experimental Findings

The evaluation compares frontier LLM defenders with heuristic strategies and a Stackelberg optimizer across attack graphs and budgets. LLM defenders are near-optimal on smaller graphs, but their evaluation and alignment with formal risk degrade with complexity and vary by budget.

  • Defender strategies: The comparison includes a Stackelberg optimum, four frontier LLM defenders, Greedy and Coverage heuristics, and an adversarially constructed poor defender.Greedy uses effectiveness but ignores topology, whereas Coverage uses graph structure but ignores effectiveness.
  • Evaluation setup: The pipeline evaluates defence portfolios from seven attack graphs across 27 graph–budget combinations, producing 216 strategies in total.Attack Graph 1 has three budget levels; the remaining graphs have four.
  • Formal-risk comparison: The optimal defender consistently achieves the lowest formal risk across all budget levels.Formal risk is computed by mapping qualitative effectiveness/confidence labels to numerical scores unavailable to LLM defenders and evaluators.
  • Defender performance: LLM defenders form a tight near-optimal cluster on the smallest graphs, while a clear gap from the optimal solution appears on larger graphs.This pattern suggests meaningful strategic-structure capture in simpler graphs but increasing difficulty with graph complexity.
  • Evaluator agreement: The optimal defender receives the highest average evaluation score, followed by Gemini, Grok, and ChatGPT, while Claude scores slightly lower despite near-optimal formal risk.Claude’s rank gap reaches +2.19 in graphs 5–7, making it the most underrated defender.
  • Budget effects: At budget level 2, the optimal defender scores 5.43 versus 4.82 for Gemini and Grok, but its advantage narrows at comfortable budgets and diminishes at tight or generous budgets.The authors suggest that constrained or highly permissive budgets make meaningful discrimination harder.
  • Evaluator robustness: Aggregated evaluation across a panel of LLM evaluators helps offset small individual self-preference effects.Gemini and Grok score their own strategies slightly higher, but the deviations are not major after aggregation.
  • Interpretation: Agreement between LLM-generated strategies and the optimization baseline occurs despite a representation mismatch because LLMs receive qualitative rather than numerical information.The comparison therefore measures convergence toward similar decisions under different representations, not shared optimization of the same objective.

5.1 Framing Effects in Evaluation Protocols

The study tests whether alternative evaluation prompts change LLM judgments of defence portfolios. Framing and scoring format materially affect rankings, with categorical judgments generally more discriminative than numerical scoring.

  • Task framing: Removing the phrase “risk reduction” and asking evaluators to assess effectiveness, budget efficiency, and attack-path coverage changes evaluation outcomes.Under this framing, ChatGPT gives the Greedy strategy a very high score, unlike the main protocol.
  • Scoring format: Adding numerical scores often produces insufficient discrimination, with multiple strategies receiving similar scores.The results remain broadly consistent with the previous experiment but compress differences between strategies.
  • Domain-knowledge effects: In the numerical-scoring experiment, evaluators consistently rank Gemini highest for the POS threat graph because its strategy prioritizes widely recognized Point-to-Point Encryption.P2PE does not optimally cover the attack paths in that graph, making this case an outlier relative to formal structure.
  • Overall finding: Evaluation framing materially affects conclusions: numerical scoring may obscure differences and amplify biases, whereas categorical judgments improve discrimination.The authors therefore advise interpreting LLM evaluations with care.

5.2 API vs Chat Consistency

API and chat-based evaluations produce broadly similar defender rankings, with only minor score variations. The section also examines self-preference and name bias, finding moderate or inconsistent self-preference but strong sensitivity to suggestive labels.

  • API vs Chat Consistency: API and chat-based evaluations produce broadly similar rankings, with only minor variations in scores.Both access modes use identical prompts over the same scenarios.
  • Name bias: The framing experiments test whether labeling a strong or poor strategy as “optimal defender” changes evaluator scores relative to anonymized baselines.The OM condition labels the poor defender as optimal, while OG labels the genuinely optimal defender as optimal.
  • Self-preference: Evaluators ChatGPT and Claude show moderate positive self-preference means of +0.33 and +0.11, while Gemini and Grok show negative shifts of −0.26 and −0.19.All evaluators have median differences of +0.0, and the pooled mean and median are both +0.0.
  • Name bias: +1.68 points is the average score increase when a poor strategy is relabeled as “optimal defender,” compared with the baseline.The score rises from 1.76 (“Bad” to “Very Bad”) to 3.44 (“Average” to “Good”).
  • Name bias: The misleadingly labeled poor strategy frequently matches or exceeds the genuinely optimal defender, reaching a 71% OM win rate in Experiment 3.Experiment 2 has a 43% OM win rate without a statistically significant mean difference; Experiment 3 reports a significant difference.

5.4 Implicit Reconstruction of Threat Models

The study tests whether LLMs can reconstruct attack graphs from progressively reduced context rather than relying only on explicit graph structure. Reconstruction quality declines as semantic context is removed, although attack-path quality remains comparatively strong under minimal context.

  • Experimental setup: LLMs reconstruct attack graphs from four variants that progressively remove scenario, graph, cost, and control information.The graph size hint remains present in every variant, while budget information is withheld from influencing topology.
  • Evaluation: Each reconstruction is evaluated by four LLM judges on structural similarity, semantic similarity, attack-path quality, and overall fidelity.The study produces 112 reconstructions and 448 evaluations across seven graphs and four variants.
  • Results: Performance declines as contextual information is removed, with the sharpest drop occurring between Variants C and D when scenario semantics disappear.The declines are −1.66 for semantic similarity and −1.55 for overall fidelity, compared with −0.96 for structural similarity and −0.56 for attack-path quality.
  • Results: Attack-path quality remains above 8.5 even under minimal context, despite larger declines in semantic similarity and overall fidelity.This pattern indicates that the evaluation distinguishes path quality from broader semantic and fidelity measures.

5.5 Effect of Removing Semantic Information

The ablations test whether evaluators use attack-graph structure rather than familiar control names. With topology retained, LLM defenders remain close to the optimum; without topology, scores shift toward simple effectiveness-based heuristics.

  • Topology retained: Control names are abstracted while complete attack-graph topology and control-to-edge mappings remain available to evaluators.Defender identities are also anonymized.
  • Topology retained: ChatGPT scores 4.83 under anonymized controls with topology retained, slightly exceeding the optimal defender at 4.77.Gemini, Grok, and Claude score 4.57, 4.54, and 4.48, while Coverage and Greedy score 3.86 and 3.77.
  • Topology removed: Removing attack-graph topology leaves only the graph name while keeping the other evaluation conditions unchanged.Controls and defenders remain anonymized, and the same evaluator panel is used.
  • Topology removed: Greedy becomes the highest-scoring defender at 5.42 without topology, while the optimal defender falls to 4.31 and LLM defenders decline overall.The reported shift is toward high-effectiveness control selection rather than structural attack-path reasoning.
  • Interpretation: With topology removed, evaluator scores collapse toward effectiveness-based heuristics, making meaningful security-strategy evaluation dependent on explicit graph structure.When topology is retained, performance remains close to the game-theoretic optimum even with anonymized controls.

6 LLM-Generated Game Solver

The paper asks LLMs to generate solvers for budget-constrained defence selection and evaluates them on attack graphs of increasing size. Generated solvers recover optimal risk when they finish, but scale substantially worse than the purpose-built baseline and may time out on large graphs.

  • Task: LLM-generated solvers implement a function that selects a control portfolio minimizing attack-path risk under direct and indirect budget constraints.The task is posed without graph-specific data and includes an attack-graph example.
  • Experimental setup: The evaluation uses 36 generated attack graphs ranging from 10 to 90 nodes, with four graphs per node count and budget B = ⌊|C|/2⌋.Graphs are grouped into small, medium, and large categories.
  • Generated algorithms: ChatGPT, Claude, and Grok formulate log-space mixed-integer linear programmes, whereas Gemini uses Dijkstra-guided branch-and-bound over control subsets.The MILP implementations use PuLP/CBC and a two-stage risk-then-cost procedure; Gemini’s approach is fast on small graphs but scales poorly.
  • Solution quality: All LLM-generated solvers achieve optimal risk when execution completes.ChatGPT times out on one 70-node graph, while Gemini times out on seven graphs.
  • Scalability: On large graphs, Grok and Claude have median running times roughly 8 times slower than the Optimal baseline.Timeouts are counted at 1,500 seconds, and the comparison excludes Coverage and Greedy because their runtimes are essentially size-independent.
  • Scalability: Execution times are one to two orders of magnitude higher than the baseline, and larger instances can fail to return solutions because of timeouts.Gemini fails on seven of twelve 80- and 90-node instances within the 1,500-second limit.

7 Discussion, limitations and recommendations

The study frames its optimizer as a controlled normative reference rather than real-world ground truth, and recommends structured, multi-model validation because LLM decisions are sensitive to framing, representation, and domain-knowledge overrides.

  • Evaluation scope: The evaluation targets LLM behaviour and robustness relative to an explicit optimization objective, not the real-world correctness of individual defence portfolios.
  • Evaluation scope: The game-theoretic optimizer provides a controlled normative reference because experts may disagree about the best budget-constrained cybersecurity portfolio.
  • Scope and assumptions: The seven scenarios derive from documented real-world attacks and capture diverse attack structures rather than reproducing every aspect of enterprise environments.
  • Scope and assumptions: Alignment with the optimization baseline decreases as attack-graph complexity increases, while the study does not address scaling attack-graph optimization.
  • Scope and assumptions: The study deliberately separates qualitative LLM reasoning from undisclosed quantitative risk mapping and mathematical optimization.
  • Recommendations: Use standardized prompts, explicit attack-graph structure, formal cross-checks, and panels of models rather than relying on a single unrestricted output.
  • Recommendations: LLM outputs should be checked against the specific system structure because models may prioritize widely recognized controls over controls optimal for the given model.
  • Recommendations: LLMs are best used as assistive tools within structured pipelines rather than standalone cybersecurity decision-makers.

8 Conclusion and future work

The paper evaluates LLM decision-making over realistic attack graphs against a game-theoretic reference. It finds that LLMs often align with explicit objectives but do not reliably optimize them, motivating complementary use with formal methods.

  • The evaluation tests LLM decision-makers on structured cybersecurity settings using attack graphs derived from real-world threats.
  • Game-theoretic optimization supplies a precise normative reference for testing whether LLM behaviour follows an explicit reasoning objective.
  • LLMs often align with the optimization objective under qualitative inputs but do not consistently optimize it.
  • LLMs approximate structured reasoning without reliably implementing it, so their role is complementary to formal methods rather than a replacement.
  • Future work includes integrating LLM candidate strategies with game-theoretic solvers, aligning models with explicit objectives, and extending evaluation to dynamic, uncertain, and framing-robust settings.
Loading 2608.20966v1…