PaperlayerView exact revision on arXiv

Source-linked AI summary

CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defense

Ryozo Masukawa, Sanggeon Yun, Raheeb Hassan, Hyunwoo Oh, SungHeon Jeong, Mohsen Imani

arXiv:2609.31893v1cs.LGcs.AIcs.CR

TL;DR

Model-free reinforcement learning for cyber defense can require extensive environment interaction, and it remains unclear what a cyber world model should represent. CyberWorld learns latent cyber dynamics from multiple representations and trains defense through imagined trajectories. Across four scoreable attack strategies, its graph variant reaches the control in 3.6k–15.8k steps, compared with millions of steps for PPO.

  • Problem

    Model-free reinforcement learning can require substantial environment interaction, and cyber world modeling must determine how to represent heterogeneous observations such as host attributes, topology, alerts, and logs.

    Source: §1 · pp. 1–2

  • Method

    CyberWorld is a Dreamer-style framework that learns latent cyber dynamics and trains defensive policies through imagined trajectories across vector, graph, textual, and multimodal representations.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across four scoreable attack strategies, graph-based CyberWorld reaches the control in 3.6k–15.8k steps, while PPO requires 2.3M–3.1M steps or fails within its 3.2M-step budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Graph structure is most consistent under topology-dependent attacks, while vector and graph representations perform comparably on most attackers.

    Source: §1 · p. 2

    Interaction efficiency does not imply equivalent wall-clock efficiency, particularly for text and multimodal variants with substantially higher inference cost.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Deep reinforcement learning has become a prominent approach to autonomous cyber defense. Existing methods are predominantly model-free and consequently require extensive environment interaction. World models provide an alternative by learning predictive dynamics and optimizing policies through imagined trajectories, yielding substantial gains in sample efficiency in robotics and embodied control. Extending this paradigm to cybersecurity raises a fundamental question: what should constitute the "world" in a cyber world model? We introduce CyberWorld, a Dreamer-style world modeling framework that learns latent cyber dynamics from vector, graph, textual, and multimodal representations of the defended network. Across all four scoreable CyberWheel attack strategies, the graph-based CyberWorld variant exceeds a strategy-agnostic control after 3.6k-15.8k environment steps, compared with millions of steps required by model-free PPO. Across representation choices, graph structure provides greater robustness under topology-dependent attacks, while simpler representations remain competitive in overall performance. Among successful runs, the number of episodes required to reach the control remains approximately constant as network size increases from 15 to 100 hosts. These results establish learned cyber dynamics as a sample-efficient and scalable basis for autonomous defense, and identify world representation as a central design axis for robustness and scalability.

1 INTRODUCTION

Cyber defense commonly relies on model-free reinforcement learning, which can require substantial interaction, while world models offer policy learning through imagined trajectories. CyberWorld asks what a cyber world model should represent and evaluates a Dreamer-style approach across representations, attack strategies, and network scales.

1 INTRODUCTION

  • Model-free reinforcement learning directly optimizes defensive policies through repeated environment interaction, which can make learning and adaptation sample-inefficient.World models instead learn predictive dynamics and optimize decisions through imagined trajectories.Source: §1 · p. 1
  • World models may retain shared network-dynamics knowledge across attacker changes, whereas a reactive policy fitted to one adversary must be relearned.The paper motivates this distinction by noting that network protocols, services, and detection mechanisms change at different rates from attacker strategies.Source: §1 · p. 2
  • The central question is what should constitute a cyber world model’s “world,” given heterogeneous observations and networks whose scale and structure vary.Defenders observe host attributes, topology, alerts, logs, and temporal histories.Source: §1 · p. 2
  • CyberWorld is a Dreamer-style world model that learns latent cyber dynamics and trains defensive policies through imagined trajectories.The framework compares per-host vectors, host graphs, combined vector-graph inputs, natural-language telemetry, and vector-text fusion with frame reconstruction.Source: §1 · p. 2
  • The study compares representations, examines vector and graph performance across network scales, and evaluates sample efficiency against model-free PPO.It also examines end-of-training reward stability to distinguish rapid initial learning from stable defensive behavior.Source: §1 · pp. 2–3
  • Across four scoreable attack strategies, graph-based CyberWorld reaches the control in 3.6k–15.8k steps, while PPO needs 2.3M–3.1M steps or fails within its 3.2M-step budget.Vector and graph representations perform comparably on most attackers, while graphs are most reliable against the topology-following attacker.Source: §1 · p. 2

2 BACKGROUND & RELATED WORKS

Prior work motivates world models as a way to learn policies through imagined trajectories and to represent heterogeneous observations. In cyber defense, earlier model-based approaches often used specified or estimated dynamics, leaving a gap for learned predictive latent models of adversarial network dynamics.

2.1 FROM MODEL-FREE REINFORCEMENT LEARNING TO MULTIMODAL WORLD MODELS

  • Dreamer-style world models learn latent dynamics and train policies primarily inside imagined trajectories, extending model-based learning beyond visual observations to language, multimodal, and structured representations.The cited background contrasts this with model-free approaches that optimize policies directly from collected experience.Source: §2.1 · p. 3

2.2 MODEL-BASED AUTONOMOUS CYBER DEFENSE

  • Earlier model-based cyber defenses used specified or estimated dynamics, while most recent deep reinforcement learning approaches learned policies directly without explicitly modeling environment evolution.CyberWorld is presented as the first Dreamer-style cyber-defense world model to learn latent dynamics capturing attacker behavior, defender interventions, and network-state evolution.Source: §2.2 · p. 3

3 CYBERWORLD

CyberWorld learns a predictive latent model from heterogeneous defender observations and trains defensive behavior within imagined trajectories. Its formulation separates shared cyber mechanisms from attacker strategy, while its encoders, recurrent dynamics, and reconstruction objectives support multiple representations of the same defender-visible information.

3 CYBERWORLD

  • CyberWorld builds a predictive model from replayed interactions and trains its policy inside model imagination, using an intermediate defender-view representation, heterogeneous encoders, and modality-specific reconstruction objectives.Its setup adapts DreamerV3 to discrete, sparse-reward enterprise defense with relational topology.Source: §3 · p. 3

3.1 PROBLEM FORMULATION

  • CyberWorld’s latent dynamics model separates shared cyber mechanisms from attacker strategy, allowing the world model, in principle, to retain mechanism information when adversary strategy changes.The shared mechanisms include protocol semantics, service configurations, exploits, and deception constraints; attacker strategy contributes a separate stochastic policy.Source: §3.1 · p. 4
  • The POMDP formulation treats telemetry as incomplete and delayed, so observations are non-Markovian and defense requires tracking a temporal belief state.CyberWorld approximates that belief with a recurrent latent state; the observation representation is therefore a dynamics-modeling choice.Source: §3.1 · p. 4

3.2 WORLD MODEL

  • Observation channels are encoded into token sequences and fused into one embedding; an RSSM maps embeddings and defender actions to latent states, and heads reconstruct observations, reward, and continuation.The architecture uses cross-attention pooling for multimodal streams, with a fixed-size observation embedding independent of host or text-token count.Source: Figure 2 · p. 4, §3.2 · p. 5
  • CyberWorld modifies reconstruction losses because persistent node attributes can reward predicting no change, while rare decoy events contribute little under uniform reward-loss weighting.The model up-weights changing state elements and balances reward-bearing steps against the remaining steps.Source: §3.2 · p. 6

3.3 BEHAVIOR LEARNING AND ONLINE TRAINING

  • The actor and critic train on trajectories imagined from replayed posterior states rolled forward through the RSSM prior, while online training alternates episode collection with model and actor–critic updates.Training starts from scratch without offline data, and reported interaction budgets include random buffer prefill.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments compare world-model representations and model-free baselines in CyberWheel, then test performance as networks grow. CyberWorld reaches the control with far fewer environment steps than PPO, while graph structure is most reliable under topology-dependent attacks.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses CyberWheel’s seven scripted attackers, with four scoreable, and compares representations and network scales under fixed interaction budgets.RQ1 and RQ2 use the 15-host network; RQ3 scales to 25, 50, and 100 hosts. The five representation arms differ in encoders and reconstruction heads while sharing latent dynamics and training configuration.Source: §4.1 · p. 7
  • The primary metric is environment steps until trailing-10 episode return first exceeds control, averaged over three seeds; runs that never cross are reported as such.PPO uses 30-episode checkpoint evaluations, while CyberWorld uses online training return.Source: §4.1 · p. 8

4.2 RESULTS

  • Across four scoreable attackers, graph CyberWorld reaches control in 3.6k–15.8k steps, while PPO needs 2.3M–3.1M steps or fails within its 3.2M-step budget.Figure 3 covers vector and graph CyberWorld, GNN-PPO, and PPO at N=15; the reported comparison concerns environment interactions, not computational cost.Source: §4.2 · p. 8, Figure 3 · p. 8
  • On three attackers, vector and graph arms are indistinguishable within seed variation; graph has the lowest state-bit and reward-head losses where representations separate.Text crosses on every scoreable attacker and is fastest on BFSExfiltration; the multimodal arm crosses every seed on the two non-BFS attackers but only one seed per BFS attacker.Source: Figure 5 · p. 8, §4.2 · p. 9
  • Among successful runs, episodes to control remain approximately constant as networks grow, but at N=100 the fixed interaction budget limits crossing frequency.Under BFSExfiltration, only graph CyberWorld reaches control at N=50 and N=100 among world-model arms; GNN-PPO’s crossing is not sustained after decoy-limit penalties accumulate.Source: §4.2 · p. 9, Figure 7 · p. 9
  • The evaluation is limited to CyberWheel with scripted adversaries and an RSSM-based model, and interaction efficiency does not establish equal wall-clock efficiency.Text and multimodal variants incur substantially higher inference cost; alternative world-model backbones are not compared.Source: §4.2 · p. 9

5 CONCLUSION

CyberWorld learns latent network dynamics and trains defensive policies through imagined trajectories. The conclusion is that dynamics modeling can reduce environment interaction, while representation choice matters for topology-dependent attacks.

5 CONCLUSION

  • CyberWorld learns predictive latent dynamics of the defended network and optimizes defensive policies through imagined trajectories.The conclusion identifies reduced environment interaction and representation choice under topology-dependent attacks as central findings.Source: §5 · p. 10

A THEORETICAL ANALYSIS OF THE PROBLEM FORMULATION

The analysis formalizes shared cyber mechanisms and strategy-dependent adversary behavior, then derives bounds on strategy shift, belief-state control, and predictive-model error. These results depend on a shared-mechanism factorization and do not establish size-independent adaptation or sample complexity.

A.1 MECHANISM INVARIANCE AND TRANSFER ACROSS ADVERSARY STRATEGIES

  • Under the shared-mechanism factorization, adversary strategies differ through their policies while the mechanism, observation model, and reward remain shared.The closed information state assumption makes the coupled process Markov; shared mechanisms assume protocol semantics, action effects, exploit preconditions, and decoy behavior do not change across strategies.Source: §A.1 · pp. 13–14
  • A change in adversary strategy cannot change the effective transition kernel in total variation by more than the corresponding change in adversary action distribution.This bound holds under Assumptions 1–2.Source: §A.1 · p. 14
  • Under partial observability, the exact filtering posterior is sufficient for defender control and evolves through a Bayes operator whose strategy dependence enters through the transition law.The optimal policy can be expressed as a function of the belief state.Source: §A.1 · p. 14
  • CyberWorld’s recurrent latent state is only an approximate predictive statistic; the analysis does not require it to recover the exact belief state.The RSSM prior and posterior play roles analogous to prediction and observation-update stages of Bayesian filtering.Source: §A.1 · p. 14
  • The model-error analysis separates error in the shared mechanism model from error in estimating the target adversary’s strategy.The appendix presents this as an analytical abstraction; CyberWorld does not explicitly learn these as separate modules.Source: §A.1 · pp. 13–14
  • The decomposed simulation bound implies that improving the shared mechanism model can benefit predictions across adversary strategies, while adaptation still depends on representing or inferring the new strategy.The guarantee is conditional on the stated assumptions.Source: §A.1 · pp. 14–15
  • Once mechanism error falls below the stated threshold, the residual-error condition requires no further reduction in that error term.The result is stated as a condition for achieving the target suboptimality.Source: §A.1 · p. 15
  • The bound has no explicit host-count term, but this does not establish size-independent adaptation or the interaction complexity of retraining a fixed policy.Larger networks may make state estimation, belief tracking, function approximation, or data collection more difficult.Source: §A.1 · p. 15

B.2 PROOF OF PROPOSITION 1

The proof establishes exact belief-state sufficiency for POMDP control, then bounds predictive-model error by mechanism and adversary-strategy error. A separate fixed-policy bound measures sensitivity to strategy shift, not the interaction cost of adaptation.

B.2 PROOF OF PROPOSITION 1

  • The exact filtering posterior is sufficient for optimal control under the stated transition model, but this does not imply that CyberWorld’s RSSM latent state recovers that posterior.The RSSM distributions are analogous to predictive and observation-conditioned latent distributions, not an established exact belief representation.Source: §B.2 · p. 16

B.3 PROOF OF THEOREM 1

  • Theorem 1’s proof combines a hybrid-kernel one-step error decomposition with a simulation-lemma argument to bound policy value error.The one-step error separates mechanism-model deviations from adversary-strategy estimation error.Source: §B.3 · p. 16

B.4 PROOF OF REMARK 1

  • The fixed-policy strategy-shift bound does not establish the interaction complexity required to retrain or adapt a model-free agent to a new adversary.Its role is to quantify sensitivity of a fixed policy and contrast that with Theorem 1’s mechanism–strategy decomposition.Source: §B.4 · p. 17

C CROSSING STATISTICS AND PERFORMANCE PROFILES AT N=15

At 15 hosts, the world-model arms reach the control much earlier and more often than PPO within their respective budgets, while GNN-PPO’s crossing profile is similar to the numeric world-model arms. The comparison also distinguishes initial crossing from post-crossing retention.

C CROSSING STATISTICS AND PERFORMANCE PROFILES AT N=15

  • The crossing measure is the first environment step exceeding the strategy-agnostic deploy_then_stop control, averaged over three seeds except four text or multimodal cells with two completed seeds.DFSImpact, BruteForce, and Inactive are omitted because their control is zero and no crossing is defined.Source: C CROSSING STATISTICS AND PERFORMANCE… · p. 17, Table 2 · p. 17
  • 92% of runs for each numeric representation reach the control within 52.5k steps, while PPO reaches 5 of 12 runs (42%) by its 3.2M-step budget.PPO’s profile remains at zero until approximately 2M steps; the text representation crosses in all but one run.Source: Figure 8 · p. 17, C CROSSING STATISTICS AND PERFORMANCE… · p. 18
  • GNN-PPO also reaches the control in 92% of runs, with a profile rising at steps similar to the numeric world-model arms; the clearest world-model advantage is post-crossing retention.The passage points to Table 3 for the retention comparison.Source: C CROSSING STATISTICS AND PERFORMANCE… · p. 18

D SENSITIVITY OF THE CROSSING CRITERION TO THE SMOOTHING WINDOW

The appendix tests whether the first-crossing result depends on trailing-return smoothing. Wider windows generally delay crossings, but the headline sample-efficiency conclusion is robust apart from a window-sensitive BFSExfiltration result.

D SENSITIVITY OF THE CROSSING CRITERION TO THE SMOOTHING WINDOW

  • Across ServerDowntime, Exfiltration, and BFSServerDowntime, widening the window from W=10 to W=50 delays crossing by at most a factor of two, against a 102–103 gap to PPO.The figure reports first-crossing step by window as mean ± sd over three seeds, with crossing-seed counts where fewer than three cross.Source: D SENSITIVITY OF THE CROSSING… · p. 18, Figure 9 · p. 18
  • On BFSExfiltration, only W=10 registers a crossing, and only for two of three seeds; wider windows do not register one.The vector arm settles close to the control after crossing, indicating behavior that is acquired but not consistently held.Source: D SENSITIVITY OF THE CROSSING… · p. 18
  • The headline conclusion is independent of the smoothing convention, while the BFSExfiltration result depends on that convention.The primary crossing metric uses trailing-10 episode return; the appendix examines W values of 10, 25, 50, and 100.Source: D SENSITIVITY OF THE CROSSING… · p. 18

E PER-REPRESENTATION LEARNING DYNAMICS AND POST-CROSSING RETENTION

The appendix expands the representation comparison with full learning curves, crossing-step results, and post-crossing retention statistics. It shows that crossing the control is not the same as sustaining it: retention varies by attacker and representation.

E PER-REPRESENTATION LEARNING DYNAMICS AND POST-CROSSING RETENTION

  • The crossing event is a transition from a random-policy return of about −104 to the control level within a few hundred episodes.Figure 10 gives learning curves for every representation and scoreable attacker; Figure 11 lists environment steps to first reach the control.Source: E PER-REPRESENTATION LEARNING DYNAMICS… · p. 19, Figure 10 · p. 19, Figure 11 · p. 19
  • Table 3 compares post-crossing statistics under different budgets: CyberWorld is measured at 52.5k steps, while PPO is measured after up to 3.2M steps.The budgets differ by a factor of 60, so the table does not bold entries for direct comparison.Source: Table 3 · p. 20
  • On Exfiltration, vector and text arms exceed the decoy-limit penalty in more than half of late episodes, while graph and multimodal arms do not.The multimodal arm holds the control with the fewest penalties on ServerDowntime and Exfiltration, but crosses later or on fewer seeds than numeric arms.Source: E PER-REPRESENTATION LEARNING DYNAMICS… · p. 20
  • On BFSExfiltration, vec+graph, text, and multimodal arms settle at the do-nothing return of 0 after a brief excursion above the control.GNN-PPO’s late median return is −3,700 on both ServerDowntime attackers and 0 on both Exfiltration attackers, with 66–237 decoy-limit penalties per 100 late episodes.Source: E PER-REPRESENTATION LEARNING DYNAMICS… · p. 20
  • On ServerDowntime, vector and graph arms hold the control in about half of post-crossing episodes, with late medians above it and fewer than two penalties per 100 episodes.The passage describes this as on par with PPO after 3M steps; Table 3 reports retention, late median return, and decoy-limit penalties under different budgets.Source: E PER-REPRESENTATION LEARNING DYNAMICS… · p. 20, Table 3 · p. 20

F SCALING ANALYSIS: CROSSING STATISTICS, LEARNING CURVES, AND COMPUTATIONAL COST FOR N ∈{25, 50, 100}

The scaling analysis reports crossing results, learning curves, and computational costs for networks of 25, 50, and 100 hosts. Among successful runs, crossing takes a similar number of episodes as network size grows, but the fixed interaction budget limits evaluation at 100 hosts.

F SCALING ANALYSIS: CROSSING STATISTICS, LEARNING CURVES, AND COMPUTATIONAL COST FOR N ∈{25, 50, 100}

  • The scaling appendix presents crossing-step results with seed counts, per-size learning curves, and computational costs for the representations.The crossing-cost fit uses runs that crossed; the full curves also include attackers for which no crossing is defined.Source: F SCALING ANALYSIS: CROSSING… · p. 20
  • At N=100, the 52.5k-step budget covers only 157 episodes, and curves remain rising as crossing becomes highly seed-dependent.The power-law fit uses only runs that crossed, so it describes successful-run cost rather than guaranteeing success.Source: F SCALING ANALYSIS: CROSSING… · p. 20
  • At N=50, crossing moves to 16–38k steps, while the absolute seed spread around the control widens because the control is 5–6× larger.At N=25, curves resemble those at N=15 with a longer prefill.Source: F SCALING ANALYSIS: CROSSING… · p. 20
  • At N=100, graph representation uses 43 GB of memory versus 10 GB for vector, with similar seconds per environment step.Figure 12 reports peak GPU memory, training seconds per environment step, and intermediate-representation size as N grows.Source: F SCALING ANALYSIS: CROSSING… · p. 20, Figure 12 · p. 21
  • GNN-PPO rises above the control early at every size, then settles far below it, reaching −9,000 to −11,000 on ServerDowntime attackers at N=50.Figure 13 shows learning curves across attackers and network sizes.Source: F SCALING ANALYSIS: CROSSING… · p. 20

G OPEN-LOOP IMAGINATION OF THE LEARNED WORLD MODEL

The visualization compares an observed held-out episode with graph-model states imagined after four observations. The model preserves coarse host-state structure, but its open-loop predictions are less reliable for hidden adversarial progress and precise reward timing.

G OPEN-LOOP IMAGINATION OF THE LEARNED WORLD MODEL

  • The 192×256 semantic canvas is an invertible rendering of four reconstruction-role bits, enabling decoded node states to be repainted for direct comparison.Hosts appear as circles, decoys as squares, and node colour encodes observable role state.Source: G OPEN-LOOP IMAGINATION OF THE LEARNED… · p. 21
  • After four observed steps, the graph arm rolls latent states forward using true defender actions without further observations; red rings mark state mismatches and yellow borders mark decoy touches.Figure 14 compares decoded model states with observed states on a held-out BFSExfiltration episode.Source: G OPEN-LOOP IMAGINATION OF THE LEARNED… · p. 21, Figure 14 · p. 23
  • The model approximately preserves observable host states but can misplace deployed decoys among reserved slots or fail to anticipate new alert activity.The topology remains fixed; slot bindings are indirectly reflected in observations, and an open-loop prior receives no new alerts.Source: G OPEN-LOOP IMAGINATION OF THE LEARNED… · p. 21
  • During fully open-loop rollout, predicted rewards often remain near zero at decoy-touch events, and large negative rewards can be misaligned with observed decoy-limit penalties.Sparse decoy-touch events depend on latent adversarial progress only partially revealed through later alerts; the figure is not evidence of pixel-perfect or event-perfect long-horizon prediction.Source: G OPEN-LOOP IMAGINATION OF THE LEARNED… · p. 21
  • Despite open-loop errors, the learned policy reaches the control in the corresponding experiments, although precise long-horizon reconstruction of every latent event is not shown to be necessary.The passage identifies prediction of adversary-dependent events as a future direction.Source: G OPEN-LOOP IMAGINATION OF THE LEARNED… · p. 23

Related papers

Loading 2609.31893v1…