Source-linked AI summary
Alignment of Autonomous Cyber Defense Agents via Adaptive Policy Shaping
Dena Mujtaba, Bharadwaj Ravichandran, Brian Hu, Arslan Basharat
TL;DR
Autonomous cyber defense agents must align reward-driven decisions with human operational priorities, a challenge complicated by security–availability trade-offs. This paper applies adaptive test-time policy shaping to balance alignment and reward without retraining. It reports a convex, scale-invariant trade-off and finds that training-time reward shaping collapses at scale, while adaptive shaping offers a practical balance.
Problem
ACD agents are often optimized for a single task reward without explicit human-defined objectives, despite trade-offs among security, confidentiality, integrity, and availability.
Source: §1 · p. 2Method
The paper shapes a fixed RL policy at test time with alignment-driven decisions and adaptively switches between reward- and alignment-driven policies based on observed state.
Source: §1 · p. 2Results
The alignment–reward trade-off is convex and scale-invariant, and training-time reward shaping collapses at scale while adaptive policy shaping offers a balanced practical approach.
Source: §5 · pp. 17–18Takeaways & Limitations
Adaptive alignment is presented as a practical way to balance reward and alignment, while deployment alignment is not guaranteed by objectives incorporated during training.
Source: §5 · p. 18The adaptive policy raises its alignment weight only after a CIA-critical host is impacted, leaving a one-step detection delay and brief unprotected window.
Source: §4.4 · p. 16
Abstract
from arXiv · showhide
Cybersecurity operators increasingly encounter sophisticated and fast-paced attacks, making autonomous cyber defense (ACD) a critical approach for supporting timely and effective decision-making. Reinforcement learning (RL)-based ACD agents have the ability to autonomously observe network states and take actions to defend and recover system assets, making them well suited for ACD. However, these agents often face critical trade-offs, such as improving network security while inadvertently degrading service availability or violating other operational priorities such as confidentiality and integrity, raising the core challenge of aligning agent behavior with human-specified goals and intent. In this work, we study the problem of aligning RL-based ACD agents with human-specified operational objectives without requiring costly retraining. We formalize alignment for ACD as a policy shaping problem, where a pretrained RL decision-making policy can be adjusted at test time using predicted attributes over agent actions. We also propose an adaptive approach that dynamically controls the alignment--performance trade-off as operational priorities shift during an ongoing cyber attack. We evaluate our method in a simulated cyber defense environment, CybORG, analyzing performance across multiple alignment objectives using the Confidentiality, Integrity, and Availability (CIA) information security triad. Our results show that adaptive test-time policy shaping can significantly improve alignment while preserving core cyber defense performance, offering a practical and scalable alternative to training-time alignment methods in RL. Our code is publicly available at https://github.com/ITM-Kitware/alignment-cage-challenge-2.
1. Introduction
Autonomous cyber defense agents must balance reward-driven defense with human operational priorities such as confidentiality, integrity, and availability. The paper proposes test-time policy shaping, including an adaptive variant, to align agents without retraining and evaluates the reward–alignment trade-off and generalization.
1. Introduction
- Misalignment can arise when agents optimize a single reward, since restoring or isolating compromised hosts can protect security while disrupting availability.The paper notes that existing ACD approaches often omit human-defined objectives and may require retraining when objectives or network configurations change.Source: §1 · p. 2
- The paper formulates alignment as test-time policy shaping that combines a fixed RL policy with an alignment-driven policy, avoiding costly retraining as priorities change.The alignment policy uses predicted action attributes, and the adaptive variant switches between reward- and alignment-driven policies based on observed state.Source: §1 · p. 2
- The approach targets the CIA triad, using Confidentiality, Integrity, and Availability as operational priorities for cyber defense.Figure 1 depicts CIA-critical Database, Front-End, and Auth servers and the RL and alignment policies.Source: §1 · p. 2, Figure 1 · p. 3
- The alignment–reward trade-off is convex and scale-invariant, varies with attacker strategy, and adaptive policy shaping offers a practical way to balance the objectives.Training-time reward shaping collapses at larger network scales, while test-time shaping generalizes across network sizes, attacker strategies, and alignment attributes.Source: §1 · p. 3
2. Related Work
Prior ACD research uses reinforcement learning in simulated adversarial environments, but agents have generally been assessed on task performance rather than alignment with human objectives. Related alignment work spans several methods and domains, while ACD-specific alignment remains relatively underexplored.
RL for Autonomous Cyber Defense
- ACD agents use defensive actions to mitigate attacks, and RL supports sequential decisions under uncertainty and changing adversarial network conditions.Research has used environments including Yawning Titan, CyberBattleSim, and CybORG, and methods such as DQN, PPO, hierarchical RL, and multi-agent RL.Source: §2 · pp. 3–4
- Despite extensive RL work in cyber defense, studies have paid limited attention to agent alignment with human values and objectives beyond task performance.Cyber defense requires balancing security objectives against operational priorities such as availability and mission continuity.Source: §2 · p. 4
Human Value Alignment
- AI alignment research addresses objectives including robustness, interpretability, controllability, and adherence to human principles, with much work focused on large language models.Methods include constitutional AI, RL from human feedback, and RL from AI feedback.Source: §2 · p. 4
RL Alignment
- RL alignment methods include reward modeling, preference learning, reward shaping, and demonstrations, while safe RL instead enforces constraints during optimization.Human values can be heterogeneous and context-dependent, and alignment in adversarial, safety-critical settings such as ACD remains underexplored.Source: §2 · p. 4
3. Methodology
The methodology evaluates adaptive test-time policy shaping in CybORG CAGE 2, using CIA objectives distinct from the environment reward. It defines CIA attribute scores from impacts on critical assets and compares alignment against performance in configurable network scenarios.
3. Methodology
- The study evaluates adaptive test-time policy shaping against training-time reward shaping, examining CIA trade-offs and generalization across network sizes and unseen adversaries.CIA attributes also serve as proxies for broader cybersecurity priorities such as financial cost, mission impact, or service disruption.Source: §3 · p. 5
3.1. Cyber Defense Simulation Environment
- The modified CybORG CAGE 2 scenario models three subnets, with red agents seeking host access and impact while blue agents monitor, remove malware, deploy decoys, and restore hosts.The environment introduces servers associated with CIA alignment attributes.Source: §3.1 · p. 5
- The environment is formalized as a POMDP whose states represent network configurations and whose defensive action set depends on hosts and per-host actions.Its observations provide partial information about host status and detected red-agent sessions; the initial belief encodes uncertainty about the starting configuration.Source: §3.1 · p. 5
- CAGE reward penalizes red-agent compromise and impact actions as well as blue-agent restores, with a maximum possible reward of 0.The reward excludes CIA-associated hosts, separating the original performance objective from the alignment objectives.Source: §3.1 · pp. 5–6
- The topology table presents scenarios of increasing scale and identifies CIA-critical hosts scored for Confidentiality, Integrity, and Availability.The enterprise subnet includes the defender host.Source: Table 1 · p. 6
3.2. Confidentiality, Integrity, and Availability (CIA) Attribute Scores
- CIA attribute drops quantify cumulative impacts on associated assets, with Confidentiality mapped to AS and DS, Integrity to AS and WS, and Availability to AS, DS, and WS.The scores adapt a resilience formulation in which each critical asset has a cost for each attribute.Source: §3.2 · p. 6
- The CIA scores invert successful-impact indicators to measure damage prevented or resolved, so prevention and remediation of red impacts contribute positively to alignment.Removing an active malicious process can preserve availability, but requires monitoring to identify the process first.Source: §3.2 · pp. 6–7
- The modified scenario allows red impacts on all hosts and removes CIA-critical hosts from blue-agent reward calculations to separate alignment from reward maximization.This setup models alignment and environment reward as distinct objectives that can compete for defensive attention.Source: §3.2 · p. 7
3.3. Autonomous Cyber Defense RL Agents
The paper trains PPO blue agents in CybORG CAGE 2, then shapes their fixed policies at test time to balance operational alignment with reward-driven defense.
3.3. Autonomous Cyber Defense RL Agents
- PPO blue agents are trained in the base CAGE 2 scenario against a red agent that alternates between CAGE-specific and CIA-critical targets.Each agent is trained for 500k steps in 200-step games.Source: §3.3 · p. 7
3.4. Alignment with Test-Time Policy Shaping
- Test-time policy shaping combines a pretrained RL policy with an alignment-driven policy as a convex combination of action distributions.The interpolation parameter α_t ranges from 0, which recovers the base RL agent, to 1, which gives full control to the alignment policy.Source: §3.4 · pp. 7–8
- The alignment policy assigns higher probability to actions whose CIA attribute scores are closer to the target vector, while retaining exploration among similarly aligned actions.Attribute scores measure each action’s one-step, cost-weighted contribution for the associated assets.Source: §3.4 · p. 8
3.5. Alignment with Adaptive Policy Shaping
The adaptive method switches between alignment- and reward-driven policies according to observed network state; a training-time reward-shaping baseline adds an alignment bonus to the CAGE reward.
3.5. Alignment with Adaptive Policy Shaping
- When any alignment-relevant host is impacted, the adaptive rule sets α_t = 1; otherwise, it sets α_t = 0 to use the reward-maximizing policy.The binary rule is deterministic and training-free, and the authors note other observation-based functions could set context-dependent α_t values.Source: §3.5 · p. 8
3.6. Alignment with Training-Time Reward Shaping
- Training-time reward shaping adds an alignment bonus to normalized CAGE reward, with the bonus weighted by λ relative to the original reward.The bonus is 1 when the selected action improves the target attribute and 0 otherwise; rewards are normalized after a ten-episode warm-up.Source: §3.6 · p. 9
3.7. Generalization and Scalability
The evaluation varies CybORG network size and red-agent strategy to assess generalization, while measuring defensive reward, attribute alignment, and resilience drop.
3.7. Generalization and Scalability
- The CybORG scenarios scale from 16 to 61 hosts, and evaluation tests robustness against red agents with varied targeting strategies.The scenarios include Meander agents targeting CIA-critical or CAGE-associated hosts and B-line agents targeting a specific attribute with topology knowledge.Source: §3.7 · p. 9
3.8. Evaluation Metrics
- Attribute alignment is measured as the fraction of decision steps on which the blue agent selects an action aligned with the target attribute.The study evaluates each CIA attribute separately alongside CAGE environment reward.Source: §3.8 · p. 10
- Resilience drop uses the Cadet et al. (2025) weighting scheme of w_C = 0.6 and w_I = w_A = 0.2, assuming equal asset costs.The authors state that these application-dependent weights are held fixed to focus on comparing alignment methods.Source: §3.8 · p. 10
4. Results and Discussion
The experiments compare baseline, training-time, test-time, and adaptive alignment across CIA priorities, adversaries, and network sizes. They show alignment–reward trade-offs, with test-time and adaptive approaches maintaining alignment more reliably as networks grow.
4.1. Baseline Blue Agent Performance
- PPO achieves the highest reward by protecting the operational server but fails to defend CIA-critical hosts, producing a resilience drop comparable to the monitor-only baseline.The Blue Monitor takes no defensive actions and provides a lower bound on performance across the figure's metrics.Source: §4.1 · p. 10, Figure 2 · p. 11
4.2. Training-Time Reward Shaping
- As λ increases, training-time shaping improves CIA protection while maintaining relatively consistent reward, but priority changes require retraining and training against one adversary can hurt performance against unseen adversaries.For Availability alignment at λ=3.0, reward is worse than baseline PPO across scenarios described in the Appendix.Source: §4.2 · p. 12, Figure 3 · p. 12
4.3. Test-Time Policy Shaping to Operational Priorities
- Across red-agent types, test-time α=1.0 achieves the strongest alignment while reducing attribute impacts and resilience drops, but at lower reward; training-time λ=3.0 often suppresses impacts with smaller reward reductions.Intermediate settings interpolate between baseline PPO and fully attribute-aligned policies.Source: §4.3 · p. 12, Figure 4 · p. 13, Figure 5 · p. 13
- Across network scales, test-time alignment rises with α and has a convex reward cost, whereas training-time alignment collapses at larger scales despite maintaining higher reward.Moderate α values offer favorable cost-benefit ratios; α=1.0 provides full compliance at a steeper penalty.Source: §4.3 · pp. 12–14, Figure 6 · p. 14
4.4. Adaptive Alignment under Performance Trade-offs
- Availability loss is harder to suppress than Confidentiality and Integrity loss, likely because Availability involves all three CIA-critical hosts.The adaptive Availability agent's suppression is more variable and less complete.Source: §4.4 · p. 14, Figure 7 · p. 15
- Adaptive α improves the reward–alignment trade-off over fixed-α test-time shaping at high α by avoiding constant reward cost, but its aggregate alignment gain over α = 0.5 is modest (≈0.52 versus ≈0.51) in the base scenario.Detection occurs one step after a CIA-tied host is compromised, leaving a brief unprotected window before the policy switches.Source: §4.4 · p. 16
4.5. Generalization to Unseen Adversaries and Network Topologies
- Training-time alignment scores reach ≈0.91 in the base scenario but degrade at the largest scales, while test-time (≈1.0) and adaptive (≈0.51-0.52) scores remain stable across four network sizes.Each experiment aggregates five agents trained with different seeds and evaluated on held-out seeds.Source: §4 · p. 10, §4.5 · pp. 16–17
- Against unseen B-line adversaries, training-time alignment scores are ≈0.69-0.74, while test-time (≈0.51) and adaptive (≈0.52) scores remain consistent across variants.Adaptive shaping achieves substantially higher reward than test-time shaping while maintaining similar alignment scores.Source: §4.5 · p. 17
5. Conclusion
The paper formulates alignment as test-time policy shaping over a fixed RL policy and compares it with training-time reward shaping and an adaptive approach. It reports a convex, scale-invariant alignment–reward trade-off and suggests adaptive shaping is a balanced approach, while training-time alignment collapses at scale.
5. Conclusion
- The paper formulates alignment as test-time policy shaping over a fixed RL policy and compares it with training-time reward shaping and adaptive policy shaping.The evaluation focuses on Confidentiality, Integrity, and Availability objectives.Source: §5 · p. 17
- Figure 9 compares alignment scores and CAGE rewards across alignment methods, red-agent variants, and four network scales.The supplied caption identifies results against B-line red-agent variants on the left and the red M-CIA agent across four scales on the right.Source: Figure 9 · p. 18, §5 · p. 18
- Training-time reward shaping collapses at scale, while the paper presents adaptive policy shaping as a balanced approach to the alignment–reward trade-off.The authors conclude that training-time alignment objectives do not guarantee alignment at deployment.Source: §5 · p. 18
Appendix A. CybORG CAGE 2 Scenario Action Space
The appendix describes the action space available to agents in the simulated CybORG cyber defense environment, including defensive actions for blue and an ordered attack chain for red.
Appendix A. CybORG CAGE 2 Scenario Action Space
- The appendix identifies the red agent’s actions as an ordered list along the attack chain.Figure S1 also shows the blue agent’s defensive actions.Source: Appendix A · p. 23
- Figure S1 shows the available red and blue agent actions in the simulated CybORG cyber defense environment.The blue agent has defensive actions, while the red agent has an ordered list of actions along the attack chain.Source: Figure S1 · p. 23, Appendix A · p. 23
Appendix B. Blue Agent Performance Across Network Scales
Across four network scales, the scenario tables report that training-time alignment deteriorates as network size increases, while test-time policy shaping maintains consistent alignment.
Appendix B. Blue Agent Performance Across Network Scales
- Training-time alignment deteriorates with increasing network size, while test-time policy shaping maintains consistent alignment across all four scales.The tables report per-attribute alignment, reward, and resilience drop for methods evaluated against the Meander and B-line red agents.Source: Appendix B · p. 23
- The tables report alignment as the fraction of steps choosing the CIA-optimal action, alongside CAGE reward and weighted resilience drop.The captions state that M-CAGE targets only the operational server, which has no CIA scoring, so only reward is reported for it.Source: Table S1 · p. 24, Table S2 · p. 25, Table S3 · p. 26
- The scaled scenarios contain 31 hosts and 300 steps, 46 hosts and 400 steps, and 61 hosts and 500 steps, each evaluated over 100 episodes.The base scenario contains 16 hosts and 200 steps over 100 episodes.Source: Table S1 · p. 24, Table S2 · p. 25, Table S3 · p. 26
Appendix C. Training-Time Reward Shaping Curves
The appendix presents training curves for CAGE reward and alignment reward across four network scales. It reports that training-time alignment reward convergence degrades as network size increases.
Appendix C. Training-Time Reward Shaping Curves
- Training-time alignment reward convergence degrades as network size increases across the four network scales.Figures S2 and S3 show the alignment-reward and CAGE-reward training curves, respectively.Source: Appendix C · p. 28
- Figure S2 plots CAGE reward during training across network sizes and alignment attributes for reward-shaping λ values from 0.5 to 3.0, with PPO as an unaligned baseline.The Blue Monitor baseline reward is annotated as BM = x ↓.Source: Figure S2 · p. 28
Appendix D. Blue Agent Gameplay Results
The appendix compares CIA, resilience, and reward traces across alignment approaches on the Base scenario and shows adaptive-agent gameplay traces in larger network scenarios. At Base scale, training-time shaping with λ = 3.0 offers strong suppression, while test-time and adaptive shaping maintain more consistent alignment as network size increases.
Training reward traces
- Figure S3 plots training alignment reward across network sizes and alignment attributes for agents trained with λ values from 0.5 to 3.0, alongside the zero-alignment-reward PPO baseline.Source: Figure S3 · p. 29
Larger-network gameplay
- Adaptive-agent gameplay traces cover Hosts 2, Hosts 3, and Hosts 4 scenarios, with 31, 46, and 61 hosts and 300, 400, and 500 steps, respectively.The traces show CIA impact and resilience evolving over episode steps.Source: Appendix D · p. 29, Figure S7 · p. 33, Figure S8 · p. 33
Confidentiality comparison
- At Base scale, training-time λ = 3.0 achieves the strongest CIA suppression at the lowest reward cost, but requires a separate training run per λ and fails at larger network sizes.Test-time and adaptive shaping use a single frozen PPO policy and scale reliably.Source: Figure S4 · p. 30
Integrity comparison
- Integrity shaping follows a similar pattern: training-time λ = 3.0 achieves the strongest suppression at Base scale, while test-time and adaptive shaping maintain more consistent alignment as network size increases.Source: Appendix D · p. 31
Availability comparison
- Availability shaping incurs a disproportionately higher reward cost because all three CIA-critical servers contribute to Availability.The passage states that protecting them is inherently harder without sacrificing task performance.Source: Appendix D · p. 32