Source-linked AI summary
Strategic Cyber Defense via Reinforcement Learning-Guided Combinatorial Auctions
Mai Pham, Vikrant Vaze, Peter Chin
TL;DR
Cyber defense needs upstream allocation of limited resources under uncertainty, but existing approaches often focus on reactive or execution-level decisions. This paper derives host-specific bundle valuations from RL Q-values and uses CAFormer to learn a constrained combinatorial auction. The learned allocations achieve competitive revenue relative to oracle and greedy baselines, remain robust to strategic misreporting, and correlate with defensive activity.
Problem
Cyber defense requires upstream allocation of limited resources across heterogeneous hosts, while prior approaches often emphasize reactive or execution-level planning.
Source: §I · p. 1Method
Results
Pearson r = 0.964, p = 0.0361 for allocation scores versus Blue defensive actions, while learned allocations are also robust under strategic misreporting and competitive with oracle and greedy baselines.
Source: D. Alignment with Cyber Objectives · p. 6, C. Effect of Truthfulness on Allocation · p. 6, §VII · p. 6Takeaways & Limitations
RL-derived values provide an interpretable basis for auction-based upstream cyber planning that reflects long-term utility and observed operational activity.
Source: A. Q-Value Distributions Across Host… · p. 5, D. Alignment with Cyber Objectives · p. 6, §VII · p. 6
Abstract
from arXiv · showhide
Cyber defense operations increasingly require long-term strategic planning under uncertainty and resource constraints. We propose a new use of combinatorial auctions for allocating defensive action bundles in a realistic cyber environment, using host-specific valuations derived from reinforcement learning (RL) Q-values. These Q-values encode long-term expected utility, allowing upstream planning. We train CAFormer, a differentiable Transformer-based auction mechanism, to produce allocations that are approximately incentive-compatible under misreporting. Rather than benchmarking against existing agents, we explore the qualitative and strategic properties of the learned mechanisms. Compared to oracle and heuristic allocations, our method achieves competitive revenue while offering robustness to misreporting. In addition, we find that allocation patterns correlate with adversarial and defensive activity, suggesting implicit alignment with operational priorities. Our results demonstrate the viability of auction-based planning in cyber defense and highlight the interpretability benefits of RL-derived value structures.
I. INTRODUCTION
The paper reframes cyber defense as upstream strategic planning under adversarial pressure and limited resources, using RL-derived valuations in a decentralized combinatorial auction. CAFormer allocates defensive bundles before execution while targeting incentive alignment and interpretable operational coordination.
Motivation
- The paper addresses the need to allocate limited defensive actions across heterogeneous enterprise assets before threats fully materialize.Existing planning approaches often emphasize reactive or tactical decisions, while the proposed layer operates upstream of episode execution.Source: §I · p. 1
Approach
- Each host is modeled as a self-interested agent with private valuations over defensive-action bundles derived from reinforcement-learning Q-values.Bundle bidding captures non-additive effects such as Analyze followed by Restore, as well as redundant or conflicting action combinations.Source: §I · pp. 1–2
- CAFormer learns permutation-equivariant, feasibility-constrained allocations and payments using differentiable mechanism design with adversarial regret minimization.The mechanism is trained to make agents or learned proxies have little benefit from misreporting valuations.Source: §I · p. 2
Contribution
- The work applies CAFormer to upstream cyber defense planning under operational budgets, extending the mechanism to a new domain.The paper presents this as a strategic planning mechanism rather than a replacement for in-episode defense behavior.Source: §I · pp. 1–2
Experimental setting
- The framework instantiates upstream planning in DARPA CAGE Challenge 2, allocating Analyze or Restore resources across heterogeneous hosts before execution.Experiments examine auction revenue and regret alongside downstream cyber-defense effectiveness.Source: §I · p. 2, Figure 1 · p. 2
II. RELATED WORK
Prior work spans centralized strategic models and autonomous cyber-defense environments. These literatures establish resource-constrained planning and realistic evaluation settings but leave distinctions between planning assumptions and operational testbeds important.
A. Cybersecurity Planning: Centralized and Sequential Approaches
- Stackelberg security games model defender-attacker interactions through leader-follower strategies but often assume centralized control and perfect observability.Source: A. Cybersecurity Planning: Centralized… · p. 2
B. Autonomous Cyber Defense and Gym Environments
- Automated cyber-defense research emphasizes realistic simulation gyms and standardized testbeds for evaluating reinforcement-learning agents under adversarial network dynamics.Examples include CyberBattleSim, CybORG, FARLAND, EIReLaND, and the TTCP CAGE challenge series.Source: B. Autonomous Cyber Defense and Gym… · p. 2
C. Attack Graphs and Vulnerability Modeling
Related approaches address vulnerability structure, sequential decision making, security auctions, and learned mechanisms from different angles. The paper positions neural combinatorial auctions as a way to combine these concerns for upstream cyber planning.
C. Attack Graphs and Vulnerability Modeling
- Attack-graph methods allocate resources toward vulnerable nodes but struggle with dynamic adversaries, action synergies, and host-level heterogeneity.Source: C. Attack Graphs and Vulnerability… · pp. 2–3
D. Decision-Theoretic Planning and Reinforcement Learning
- MDPs, POMDPs, and reinforcement learning support long-term or adaptive cyber defense but often rely on centralized planning and rarely address upstream pre-attack resource allocation.Source: D. Decision-Theoretic Planning and… · p. 3
E. Auctions and Mechanism Design for Security
- Security auctions provide decentralized incentive-compatible resource coordination, but prior cyber applications addressed intrusion detection or DoS defense rather than strategic pre-allocation.Source: E. Auctions and Mechanism Design for… · p. 3
F. Differentiable Mechanism Design
- Differentiable mechanism-design models learn revenue- or regret-oriented auctions, while CAFormer adds permutation invariance, combinatorial feasibility, and adversarial regret training.Source: F. Differentiable Mechanism Design · p. 3
- The paper presents neural auctions as a first application to upstream cyber defense planning, combining simulation-derived valuations with constraint-aware training.Source: F. Differentiable Mechanism Design · p. 3
III. PROBLEM FORMULATION
The problem formulation represents upstream defensive-action allocation as a constrained combinatorial auction over hosts and bundles. It specifies probabilistic assignments, payments, feasibility, approximate incentive compatibility, and voluntary participation.
III. PROBLEM FORMULATION
- Each host receives a private valuation function over bundles from a finite defensive action set, representing expected defense impact before an attack episode.Source: §III · p. 3
- Hosts report bid vectors over predefined non-empty bundles, and the planner chooses a probabilistic allocation distribution together with payments.Source: §III · p. 3
- Feasibility requires each host to receive at most one bundle and each individual action to be assigned to at most one host across all bundles.Source: §III · p. 3
- Approximate incentive compatibility is pursued by minimizing ex post regret, while individual rationality requires each host’s expected utility to remain nonnegative.Source: §III · p. 3
IV. VALUATION GENERATION
The paper generates host-specific defensive valuations from a PPO-based Blue agent in DARPA CAGE Challenge 2, using critic Q-values to represent long-term consequences and normalized inputs for auction training.
IV. VALUATION GENERATION
- DARPA CC2 provides a 13-host manufacturing-plant defense setting with persistent phishing and lateral-movement attacks over fixed episode horizons.The evaluation uses Scenario 2, scripted Red policies, and a Blue agent tasked with maintaining resilience without disrupting critical assets.Source: §IV · p. 3
- The valuation pipeline uses the Team CardiffUni PPO agent, whose host-specific decoys, scan-memory vectors, and fallback behavior provide a realistic downstream policy.The agent won first place in DARPA CAGE 2 and supplies the policy context for upstream auction planning.Source: §IV · p. 3
A. Q-values Estimation
- Critic Q-values replace sparse immediate rewards because they encode both immediate consequences and downstream effects under adversarial dynamics.The paper computes Q_i(s, a) through value-plus-advantage decomposition, reducing variance while capturing longer-term decision consequences.Source: A. Q-values Estimation · p. 4
- Global min-max normalization maps valuation inputs to [0, 1], stabilizing optimization and constraining adversarial misreports to the same domain.The transformation preserves relative Q-value rankings and avoids poorly scaled values that could produce invalid economic interpretations or hinder convergence.Source: A. Q-values Estimation · p. 4
B. Valuation Model
The valuation model assigns each host bundle values from learned per-action Q-values and adds structured curvature to represent complementarities, redundancy, and preference uncertainty.
B. Valuation Model
- Each host defines a private valuation over bundles of Analyze, Remove, and Restore, whose values are derived from learned per-action Q-values.These core actions are consistently defined across hosts and directly comparable.Source: B. Valuation Model · p. 4
- Deploy Decoy, Activate Decoy, and Sleep are excluded because they are runtime-dependent or have ambiguous utility for offline planning.The first two depend on host context, while Sleep is considered strategically limited in the valuation model.Source: B. Valuation Model · p. 4
- The model distinguishes additive, submodular, and supermodular bundles to represent independent contributions, diminishing returns, and synergistic action combinations.The interaction strength is controlled by θ, with its sign determining submodular or supermodular behavior.Source: B. Valuation Model · p. 4
- Quadratic curvature approximates pairwise bundle interactions in ∼O(|S|^2), providing a computationally efficient surrogate for explicitly modeling each interaction.The same form captures redundancy costs in submodular settings and compounded synergy in supermodular settings.Source: B. Valuation Model · p. 4
- A multiplicative random perturbation models uncertainty from partial observability, stochastic dynamics, imperfect learning, and host heterogeneity.This stochastic curvature is intended to make auction training robust to host-level preference noise while preserving the base Q-value structure.Source: B. Valuation Model · p. 4
V. MODEL ARCHITECTURE AND TRAINING PROCEDURE
CAFormer maps Q-value-derived bundle valuations to probabilistic feasible allocations and normalized prices, while training balances revenue against regret from misreporting.
Architecture
- CAFormer predicts probabilistic allocations over feasible action bundles and normalized prices representing fractions of reported bids.The number of non-empty bundles is M = 2^|A| − 1, with n host agents.Source: §V · p. 4
Training objective
- Training minimizes revenue loss using normalized valuations, preserving ordinal preferences while supporting bounded optimization and interpretability.Normalization constrains valuations to [0, 1].Source: §V · pp. 4–5
Incentive alignment
- Regret measures the utility gain from misreporting and is estimated by differentiable gradient ascent over feasible perturbed valuation vectors.The regret term is combined with revenue through γ, which controls their trade-off.Source: §V · p. 5
Objective
- The revenue objective promotes high-value, cost-effective defensive allocations while remaining adaptable to utility, budget, or fairness objectives.Revenue is defined as the total value-weighted payment charged for defensive bundles.Source: §V · p. 5
A. Q-Value Distributions Across Host Types
The paper uses RL Q-values as long-term valuation proxies and finds a stable distributional structure across heterogeneous host types.
A. Q-Value Distributions Across Host Types
- Q-value distributions show a stable structure across host types, supporting their use as inputs for mechanism design.The authors interpret these values as direct measures of long-term utility from an RL-trained agent.Source: A. Q-Value Distributions Across Host… · p. 5
- Q-values allow the auction layer to inherit strategic foresight from lower-level reinforcement learning systems.This connects long-term utility estimation with upstream auction-based decision making.Source: A. Q-Value Distributions Across Host… · p. 5
B. Training Dynamics
The experiments compare learned, oracle, and heuristic allocation behavior while examining whether Q-value-driven allocations reflect operational activity.
B. Training Dynamics
- The oracle maximizes social welfare with deterministic winner determination and VCG payments, providing a truthful strategic reference rather than a revenue-maximizing benchmark.The paper emphasizes allocation behavior under truthful and strategic inputs.Source: B. Training Dynamics · p. 5
- Q-value distributions are heavily right-skewed, centered near mild positive values, and extend to negative values as low as −11.The negative tail indicates that some bundles or actions may be undesirable because of redundancy or risk.Source: Figure 2 · p. 5
- CAFormer and oracle-VCG training curves converge for revenue and regret, with oracle revenues of 2.676 for all hosts, 2.6752 for enterprises, and 2.677 for users.Regrets are close to zero after 10,000 iterations, with results averaged over three runs and standard deviation ≤0.001.Source: Figure 3 · p. 5
C. Effect of Truthfulness on Allocation
Allocation behavior remains structured under misreporting and correlates with defensive activity, while the conclusion identifies both interpretability benefits and a centralized-Q-value limitation.
C. Effect of Truthfulness on Allocation
- User hosts are frequently prioritized for aggressive actions, consistent with Q-values reflecting long-term strategic impact and early-stage containment needs.The paper connects this pattern to findings that User hosts can serve as stepping stones toward more privileged targets.Source: C. Effect of Truthfulness on Allocation · p. 6
- Under strategic misreporting, allocations become more diffuse but preserve host ordering, indicating structural robustness without full strategyproofness.Truthful allocations concentrate on Remove and {Analyse, Remove}, whereas greedy allocation overuses {Analyse, Remove, Restore}.Source: Figure 4 · p. 6, C. Effect of Truthfulness on Allocation · p. 6
D. Alignment with Cyber Objectives
- Allocation scores correlate strongly with Blue defensive activity across hosts (Pearson r = 0.964, p = 0.0361), while Red activity shows a weaker nonsignificant association.The result suggests alignment with observed operational activity despite no explicit mission-criticality metadata.Source: D. Alignment with Cyber Objectives · p. 6, Table I · p. 6
VII. CONCLUSION
- The framework offers interpretable RL-derived valuations but assumes a fully trained downstream agent and treats centralized Q-values as proxies for private utilities.The authors suggest host-decentralized training as a way to narrow this modeling gap.Source: §VII · p. 6