Source-linked AI summary
Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agents
Sabrina Saika, Yinuo Du, Aritran Piplai
TL;DR
Cyber attack policies are commonly evaluated within one simulator, leaving transfer across environments with differing state and action representations unclear. The paper separates state alignment from action translation to transfer policies without retraining. Transfer is feasible but conditional on state alignment, with the encoder especially useful when schemas diverge.
Problem
Cyber attack policies are typically developed and evaluated within one simulator, leaving unclear whether they transfer across environments with different state and action representations.
Source: §1 · pp. 1–2Method
The framework separates state mapping from action mapping, using a kill-chain interface and a modified DAPN encoder to transfer policies without retraining.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5Results
Transfer is conditional on state alignment: under a large schema gap, DAPN wins 100% while feature engineering scores 0%, whereas feature engineering is competitive under a narrow gap.
Source: §6 · p. 7, §6.2 · p. 10Takeaways & Limitations
The translator alone is sufficient for similar feature schemas, while the encoder recovers useful behavior when schemas diverge and naive transfer collapses.
Source: §7 · p. 11Emulation introduces exploit-version mismatches, indirect or delayed outcome observation, and non-deterministic service behavior; evaluation on branching attack topologies remains future work.
Source: §7 · p. 12
Abstract
from arXiv · showhide
Cyber attack agents are typically trained and evaluated within a single simulator, making it unclear whether learned policies transfer beyond the environments in which they were developed. This limitation hinders both deployment and fair comparison, as cyber simulators differ substantially in their state representations, observation models, and action spaces. In this paper, we study policy transfer across cyber environments and argue that simulator-to-simulator and simulator-to-real transfer can be viewed as instances of the same underlying alignment problem. We propose a framework that separates state alignment from action translation, enabling a policy trained in one environment to operate in another without retraining. We evaluate transfer across four cyber platforms, CyberBattleSim, NetSecGame, CyberWheel, and NASim, including emulated deployments in NASim. Our experiments show that zero-shot transfer is feasible, fully preserving source-policy performance in closely aligned environments and achieving 45.2% win rates when transferring policies whose source performance is 60.5%. In emulated virtual machine environments, transferred policies exhibit a Jensen-Shannon divergence of 0.085 from native policies, indicating strong behavioral similarity. Code and benchmarks are available at: https://anonymous.4open.science/r/RL-Transfer-between-env-4F47/.
1 Introduction
The paper frames transfer across cyber environments as a way to address agents’ brittleness and the difficulty of comparing or deploying policies trained in isolated simulators. It proposes treating sim-to-sim and sim-to-real transfer as a shared alignment problem, separating state understanding from environment-specific action execution.
1 Introduction
- LLM agents may adapt to novel attack scenarios but can abruptly refuse mid-workflow, while RL agents are controllable yet brittle.The paper identifies this unpredictable compliance as a reliability concern for autonomous red-teaming and penetration testing.Source: §1 · pp. 1–2
- Cyber agents face a deployment challenge, and simulator-specific assumptions make cross-environment performance comparisons problematic.The paper also notes that building a high-fidelity replica for each target network is prohibitively expensive.Source: §1 · pp. 1–2
- RL agents are controllable but brittle: policies can overfit network topology and degrade when it changes.Their fixed abstract state and action spaces also lack direct counterparts in real deployments.Source: §1 · p. 2
- Organizations cannot simply deploy a simulation-trained policy or compare it with an agent trained in another simulator because their state and action representations differ.The paper frames both needs as moving a learned policy between environments without retraining from scratch.Source: §1 · p. 2
- The paper treats sim-to-sim and sim-to-real transfer as alignment across environments, with partial target knowledge distinguishing sim-to-real.It evaluates zero-shot transfer, transferable policy aspects, and transfer under partial observability across CyberBattleSim, NetSecGame, CyberWheel, and NASim.Source: §1 · p. 2
2 Related Work
Prior cyber agents and simulators remain tied to their backend designs, limiting transfer and cross-environment comparison. The paper connects policy transfer with sim-to-real and representation-alignment research to frame cross-simulator transfer as a shared-representation problem.
2.1 Cyber Reinforcement Learning Agents
- Cyber simulators differ in observation dimensionality and action granularity, leaving results siloed and agents or interfaces unable to carry across environments.This section situates RL agents alongside heuristic and LLM agents [10] and notes the development of many simulators.Source: §2.1 · p. 3
2.2 Policy Transfer Across Environments
- Policy transfer asks whether a policy learned in one environment can be reused in another, and the paper addresses mismatched action and observation spaces through DAPN alignment.The approach draws on domain-adversarial alignment and the shared kill-chain progression found across simulators.Source: §2.2 · p. 3
2.3 Simulation-to-Real Transfer
- Cyber sim-to-real transfer is costly and risky to evaluate directly, so the paper uses controllable sim-to-sim transfer to gain insights about it.The broader sim-to-real challenge is also noted in robotics, autonomous driving, and control [1].Source: §2.3 · p. 3
2.4 Representation Alignment
- Representation alignment places inputs from different domains in a shared feature space, which the paper applies to make simulator observations comparable for one policy.The figure depicts transfer through state alignment and action translation.Source: §2.4 · p. 3, Figure 1 · p. 4
3 Methodology
The method decomposes transfer between environments with mismatched states and actions into state mapping and action mapping. A deterministic kill-chain interface handles structural differences, while a modified DAPN encoder addresses remaining distributional differences.
3.1 Problem Statement
- The transfer problem is to reuse a competent source policy in a target simulator despite generally different state and action spaces.The framework maps target states into the source policy’s state space and maps source actions into executable target actions.Source: §3.1 · p. 4
3.2 Kill-Chain Intent Interface
- The kill-chain intent interface constructs deterministic observation and action mappings without modifying the underlying simulators.It projects observations into a fixed vector organized by kill-chain stage per host, then maps shared host-selection intents and no-op into simulator-native actions.Source: §3.2 · pp. 4–5
3.3 DAPN Encoder
- Even after structural mapping, state values can differ because simulators have different network sizes, discovery rates, and reward scales.The modified DAPN encoder aligns the variable-distribution kill-chain stream, while semantically identical goal flags bypass adversarial alignment; reconstruction helps prevent latent collapse.Source: §3.3 · p. 5
- The encoder uses adversarial domain confusion, reconstruction, and latent-mean alignment, and its checkpoint is selected at chance discriminator accuracy.Pretraining uses random-policy observations from each simulator, without policy trajectories or target-domain rewards.Source: §3.3 · pp. 5–6
4 Environments
The evaluation transfers a policy from CyberWheel to simulated targets and uses NASim emulation as a real-world proxy. The environments differ in schema alignment, training cost, and task horizon.
4 Environments
- CyberWheel is the source domain, with training efficiency, a compact kill-chain observation, and YAML configurability.The evaluation transfers policies from this source to two simulated targets and includes a dockerized emulation environment as a proxy.Source: §4 · p. 6
- CyberBattleSim presents the larger domain gap, with high-dimensional, weakly structured observations and only partially aligned actions.It is sample-inefficient to train but has a short task horizon once a policy exists.Source: §4 · p. 6
- NetSecGame has a smaller domain gap because its observation and action spaces are closer to CyberWheel, but its task is harder and longer.Its task requires discovery followed by impact over a wider action set.Source: §4 · p. 6
- NASim emulation instantiates network nodes as Docker containers or VMs, replicates vulnerabilities on live services, and executes exploits via Metasploit.The paper presents it as a practical real-world proxy and defers full emulation results to future work.Source: §4 · p. 6
5 Experimental Setup
The setup evaluates PPO policies trained in one simulator and applied directly to another without retraining, under transfer conditions that vary state and action alignment.
5 Experimental Setup
- Four conditions compare random-policy and zero-shot transfer baselines with feature-engineering transfer and DAPN transfer.The conditions test index-aligned padding, feature alignment alone, and feature alignment plus distributional-gap closure.Source: §5 · pp. 6–7
- PPO source policies are trained in MA and applied directly to MB without retraining.Source: §5 · p. 6
- Table 2 lists the kill-chain phase mappings used by ψ for CW→CBS, CW→NSG, and CW→NASim transfer pairs.Source: Table 2 · p. 7
6 Results
DAPN enables zero-shot transfer across both narrow and large simulator gaps, while feature engineering succeeds only on the narrow gap. Its aligned state representation also brings transferred policies’ NASim action distributions closer to the emulator policy.
6 Results
- DAPN yields latent centroid distance 28.05, but the raw-feature distance of 68.44 is from a different space and is not directly comparable; visual overlap is primary evidence.The t-SNE projection shows substantial overlap between the encoded CyberWheel and NetSecGame observations.Source: §6 · p. 7, Figure 2 · p. 8
6.1 Can offensive cyber policies transfer zero-shot across different environments without retraining? (RQ1)
- DAPN wins 45.2% on narrow-gap NSG versus 47.4% for feature engineering, and 100% on large-gap CBS versus 0% for feature engineering.The NSG results use 3 seeds; the CBS chain-12 result covers 20 episodes.Source: §6.1 · pp. 8–9, Table 3 · p. 8, Table 4 · p. 9
6.2 What aspects of a policy (e.g. states, actions) are actually transferable across environments? (RQ2)
- DAPN narrows the CW–NSG performance gap to 15.3 points, a 70% reduction, while feature engineering drops 51.7 points from 99.1% CW to 47.4% NSG.The source-to-target results indicate that the encoder’s representation transfers more evenly across domains than feature engineering.Source: §6.2 · pp. 9–10
- On CBS, feature engineering alone makes observations shape-compatible but not distribution-compatible with the CW-trained policy; it emits only no-ops, averaging 0.00 nodes owned and 19 return, while state-distribution alignment enables transfer.Source: §6.2 · p. 10
- Transferred policies are closer to NASim emulator behavior: JS is 0.087 for CW and 0.085 for NSG, versus 0.111 for original CW, which does not win.The comparison uses action-type distributions over winning trajectories.Source: Figure 3 · p. 11, §6.3 · p. 11
7 Conclusion and Future Work
The proposed zero-shot transfer method combines feature translation and domain-adaptive encoding, with the encoder especially important when schemas diverge. Direct emulation remains challenging because real environments add practical obstacles that simulators abstract away.
7 Conclusion and Future Work
- The method combines a kill-chain-based feature translator with a domain-adaptive encoder to transfer policies across cybersecurity simulators without retraining.Source: §7 · p. 11
- When source and target feature schemas are similar, the translator alone is sufficient; when they diverge, the encoder recovers useful behavior where naive transfer collapses.Source: §7 · p. 11
- Direct emulation remains an open challenge because exploit-version mismatches, delayed or indirect action feedback, and nondeterministic service behavior are abstracted away by simulators.Source: §7 · pp. 11–12