PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask the Expert: LLM-Guided Reinforcement Learning for Autonomous Cyber Defense

Fernando Martinez, Abhishek Satyam, Tao Li, Junaid Farooq, Ying Wang, Juntao Chen

arXiv:2610.09337v1cs.CRcs.LG

TL;DR

Autonomous cyber defenders learn under delayed, partial observations and large action spaces, while direct runtime LLM use raises latency and trust concerns. Ask the Expert queries an LLM intermittently during training for host-level advice and shapes PPO rewards; across CC1 and CC2 and both attacker types, it improves sample efficiency over PPO and evaluated PBRS baselines while retaining the strongest terminal mean.

  • Problem

    Autonomous cyber defense requires learning under delayed, partial observations and large action spaces, while global rewards provide weak tactical feedback and runtime LLM use raises latency and trust concerns.

    Source: §I · p. 1

  • Method

    Ask the Expert intermittently queries an LLM about hard states involving newly spreading compromises, then distills host-level advice into persistent tiered reward shaping for training PPO.

    Source: §I · p. 1, D. Tiered LLM-Guided Reward Shaping · p. 3

  • Results

    Across CC1 and CC2 and both attacker types, Ask the Expert improves sample efficiency over PPO, beats evaluated PBRS variants throughout reported curves, and has the strongest terminal mean.

    Source: §VII · p. 6

  • Takeaways & Limitations

    The results support training-time semantic guidance as an asymmetric approach that leaves deployment lightweight and fully policy-based.

    Source: §VI · p. 6

    The gap to PBRS narrows during late training in some settings.

    Source: §VI · p. 6

Abstract

from arXiv · show

Policy-based reinforcement learning (RL) approaches have produced promising results for autonomous cyber defense; however, they are sample-inefficient in settings where defenders must respond under delayed, partial observations with actions from large action spaces. While large language models (LLMs) may reason semantically about security state space, high latency and trust assumptions prevent attractive in-line deployment models. We introduce Ask the Expert, a training-time guidance framework which first summarizes hard cyber-defense states, then intermittently queries an LLM for host-level defensive recommendations via a constrained action interface, and finally transforms those recommendations into tiered reward shaping for use with PPO. Because the LLM is discarded after training, deployment is a pure RL policy. Across TTCP CAGE CC1 and CC2 and both attacker types, this asymmetric design improves sample efficiency over PPO and outperforms the evaluated potential-based reward shaping (PBRS) baselines, while retaining the strongest terminal mean and requiring no LLM dependency at deployment time.

I. INTRODUCTION

Autonomous cyber defenders must respond to lateral movement with partial observations, tight time constraints, and large action spaces, while learning from rewards that provide weak tactical feedback. Ask the Expert uses intermittent LLM guidance during training to shape rewards, then discards the LLM before deployment.

Motivation

  • Defenders face lateral-movement campaigns that reuse credentials and pivot across hosts, while acting under tight time constraints and incomplete network observability.They must choose among semantically dense actions using local alerts and rewards that are dense but tactically weak.Source: §I · p. 1
  • Direct RL exploration can degrade services before reaching reasonable behavior, while global network-health rewards provide little tactically valid feedback for granular actions.The paper uses empirically inspired training-time shaping rather than claiming policy invariance from PBRS.Source: §I · p. 1
  • Online LLM control would incur unacceptable inference latency and trust-boundary risks, motivating an asymmetric training regime that leaves the deployment policy free of LLM calls.The paper points to SCORER and CORAL [14] as related asymmetric training regimes.Source: §I · p. 1

Contributions

  • The framework intermittently samples hard states where the defender first observes a newly spreading compromise, turning zero-shot LLM remediation advice into persistent training-time reward shaping.After training, the LLM is discarded and deployment uses a fully autonomous policy.Source: §I · p. 1
  • Recent-frontier prompting stores hard states involving newly spreading compromises, prioritizes recent severe failures, and caps prompts focused on any one spreading host.Host-level LLM recommendations become persistent tiered priors over remediation actions.Source: §I · p. 1
  • Ask the Expert improves sample efficiency over PPO and observation-based PBRS baselines on CAGE CC1 and CC2 against two attacker types.The framework uses LLM guidance only during training, with no deployment-time LLM calls or added inference latency.Source: §I · p. 1

II. RELATED WORK

Prior work applies RL and LLMs to network defense, training signals, and reward design, but Ask the Expert distinguishes itself by using intermittent host-level LLM advice only as training-time reward bonuses.

RL for autonomous cyber defense

  • Earlier cyber-defense approaches use heuristic reward shaping or action masking, whereas Ask the Expert turns intermittent advice into dynamically tiered reward bonuses rather than fixed hand-designed rules.Its recent-frontier sampler selects LLM prompt candidates, not TD updates as in prioritized selection.Source: §II · p. 2

Auxiliary signals and structured advice for RL

  • Prior asymmetric methods include Policy Shaping, SCORER, CORAL [14], and Wang et al.’s LLM demonstrations and reward shaping; Ask the Expert uses no demonstrations or deployment-time LLM calls.Source: §II · p. 2

LLMs for RL and Security

  • LLMs have generated reward code, subgoals, progress rewards, reward reshaping, and hierarchical control,,,,,, while runtime security use raises latency and trust concerns –.Ask the Expert instead uses training-time advice as reward bonuses and discards the LLM at deployment.Source: §II · p. 2

III. PROBLEM FORMULATION

The paper models intrusion response as a partially observed interaction between a Blue defender and a concurrent Red attacker. PPO acts in the full verb–host action space, while the LLM interface is limited to host-level remediation.

A. Modeling Network Defense as a POMDP

  • The defender receives delayed, partial observations based on host alerts and network telemetry, not the true global state.The defender never observes Red’s internal state or intended future plan during training or deployment.Source: A. Modeling Network Defense as a POMDP · p. 2

B. Threat Model

  • The modeled threat is post-compromise lateral movement in CAGE CC1 and CC2, with Red beginning each episode at User0.Source: B. Threat Model · p. 2
  • B_lineAgent attempts a direct pivot toward the operational server, while Blue reacts only to defender-visible aftermath signals.Source: B. Threat Model · p. 2
  • RedMeanderAgent sweeps subnets before penetrating inward, while Blue reacts only to defender-visible aftermath signals.Source: B. Threat Model · p. 2

C. Action Space and Semantics

  • The defender’s action space is a large discrete verb–host space, and PPO acts over the entire environment action space.Source: C. Action Space and Semantics · p. 2
  • The LLM-facing interface exposes only a smaller host-level remediation subspace, simplifying the language interface while preserving the learned policy’s full fidelity.Source: C. Action Space and Semantics · p. 2

D. Reward and Objective

The environment’s reward reflects operational compromise costs and penalties for unnecessary disruption, but hidden tactical value can make immediate actions hard to assess. The training framework adds semantic guidance without changing environment dynamics or the deployment policy class.

D. Reward and Objective

  • The environment penalizes compromised hosts according to operational criticality, while aggregate reward captures harm from unnecessarily disrupting benign services.Source: D. Reward and Objective · p. 2
  • Hidden penalties obscure immediate action value, leaving rudimentary agents to hesitate during active pivoting or blindly disrupt critical services or subnets.The framework adds semantic training-time guidance without changing environment dynamics or the deployment-time policy class.Source: D. Reward and Objective · p. 2

A. Framework Architecture

  • Three training-time modules encode defender-visible telemetry, sample recent spreading-state transitions for LLM queries, and convert host-specific advice into tiered bonus weights.PPO selects actions; the LLM affects training only through reward shaping and is discarded after training.Source: A. Framework Architecture · p. 2

B. Semantic State Encoder

  • The deterministic semantic encoder uses defender-visible reward-breakdown telemetry and previously known compromised hosts to summarize compromised, newly detected, and clean hosts.The resulting summary is compact for a general-purpose language model and uses no simulator hidden state.Source: B. Semantic State Encoder · pp. 2–3
  • Figure 1 shows spreading hard states stored in a candidate bag, sampled from a recent frontier, and paired with host-level advice that shapes rollout-end rewards before PPO advantage computation.Deployment remains pure RL with no LLM calls.Source: Figure 1 · p. 3

C. Hard-State Queue and LLM Querying

Ask the Expert keeps a fixed-capacity queue of transitions where compromises are spreading, then samples recent failures for intermittent LLM queries. The sampler prioritizes severe reward drops while limiting any one host’s prompt share.

C. Hard-State Queue and LLM Querying

  • The candidate bag stores spreading-state transitions and drops its oldest entries when it exceeds its fixed capacity.Reward change is retained as LLM context, not used to decide whether an entry is stored.Source: C. Hard-State Queue and LLM Querying · p. 3
  • Each query selects recent failures ranked by the most negative reward deltas, with at most three prompt slots per spreading host.The sampler selects up to n_s entries subject to the host cap.Source: C. Hard-State Queue and LLM Querying · p. 3
  • Prompts combine network context and prior suggestion outcomes with semantic state summaries, PPO actions, and qualitative outcome tags.The LLM returns recommendations from a constrained host-level remediation vocabulary; invalid outputs are rejected, and prompt doctrine can reflect attacker structure.Source: C. Hard-State Queue and LLM Querying · p. 3

D. Tiered LLM-Guided Reward Shaping

LLM advice becomes per-host, weighted action tiers that supplement logged rewards during training, without changing PPO’s action selection. Advice is merged, refreshed, and eventually expires.

D. Tiered LLM-Guided Reward Shaping

  • A valid LLM suggestion becomes a host’s high-weight primary tier, while semantically related actions become lower-weight fallbacks.Hosts start with weak default tiers over conservative remediation actions.Source: D. Tiered LLM-Guided Reward Shaping · p. 3
  • After each rollout and before GAE, matched PPO actions receive a bonus added to their logged reward; unmatched actions keep the original reward.The bonus is capped at β = 0.01, applies only at logged spreading-state timesteps, and cannot override PPO’s action.Source: D. Tiered LLM-Guided Reward Shaping · p. 3

E. Policy Optimization and Advice Merging

  • PPO computes generalized advantage estimates from shaped rewards and updates its policy and value network each iteration.The policy continues to choose actions using its standard policy and value network.Source: E. Policy Optimization and Advice… · p. 4
  • New host policies merge into a global advice dictionary; stale policies expire to weak defaults, and unaddressed spreading hosts receive a full-weight Restore tier.This keeps active advice focused on the current failure frontier.Source: E. Policy Optimization and Advice… · p. 4

V. EXPERIMENTAL EVALUATION

The evaluation uses two partially observed TTCP CAGE challenges, two attacker types, and PPO with observation-based PBRS as a reward-shaping control. Ask the Expert’s training loop periodically queries an LLM and applies the resulting advice during PPO training.

A. Experimental Details

  • CC1 and CC2 model 13-host networks over 100-step episodes, with partial observations, 54- and 145-action spaces, and evaluations against two attacker types.CC1 includes Misinform; CC2 replaces it with service-specific Decoy actions.Source: A. Experimental Details · p. 4
  • PPO uses an MLP policy with learning rate 3 × 10^-4, 2,048 steps per rollout, batch size 128, six epochs, γ = 0.99, and clip range 0.2.The observation-based PBRS control varies λ over {0.1, 0.3, 1.0}; Ask the Expert uses β = 0.01 and queries every 10 rollouts after a 20-rollout warmup.Source: A. Experimental Details · p. 4
  • During rollouts, the algorithm encodes reward telemetry into compromised, spreading, and clean host sets, then records semantic summaries and hard-state positions.The hard-state positions correspond to timesteps with spreading hosts.Source: A. Experimental Details · p. 4
  • At scheduled intervals, the algorithm samples recent-frontier entries, builds a prompt, and merges validated LLM advice into host policies.The schedule uses the warmup and update cadence; stale policies expire and spreading hosts not addressed by the response receive a full-weight Restore tier.Source: A. Experimental Details · p. 4
  • After advice is appended to the history, PPO computes GAE from shaped rollout rewards and updates its policy and value network.The rollout buffer is then cleared.Source: A. Experimental Details · p. 4
  • The LLM interface exposes host-level remediation actions, while PPO retains the full environment action space; evaluations are deterministic over 20 episodes.The LLM vocabulary is narrower in CC2 and additionally includes Misinform in CC1.Source: A. Experimental Details · p. 4

B. Performance Results

In CC1, Ask the Expert improves early learning over PPO and outperforms the evaluated PBRS variants, with the strongest terminal means reported across the attacker settings. Figure 2 plots checkpoint rewards for both attackers.

B. Performance Results

  • Across CC1 and CC2, Ask the Expert exceeds every evaluated PBRS variant at reported checkpoints; against B_lineAgent in CC1, its 300k mean is −22.3 versus PPO’s −25.9 and PBRS’s −34.4.Figure 2 shows the CC1 checkpoint evaluation rewards for both attacker types.Source: Figure 2 · p. 5, B. Performance Results · p. 5
  • Against RedMeanderAgent in CC1, Ask the Expert reaches −27.9 at 100k versus PPO’s −36.8 and PBRS’s −93.8, and finishes at −9.7 versus −27.7 and −18.0.The comparisons are respectively against PPO and the best PBRS mean.Source: B. Performance Results · p. 5
  • In CC2 against B_lineAgent, Ask the Expert reaches −61.3 at 100k versus PBRS’s −88.1 and PPO’s −89.7, then finishes at −17.9 versus −24.7 and −21.5.The terminal comparisons are respectively against the best PBRS mean and PPO.Source: B. Performance Results · p. 5
  • In CC2 against RedMeanderAgent, Ask the Expert reaches −98.8 at 100k versus PBRS’s −132.7 and PPO’s −139.9, and finishes at −28.6 versus −34.2 and −30.4.The terminal comparisons are respectively against PBRS and PPO.Source: B. Performance Results · p. 5

C. Mechanistic Analysis of Frontier Sampling

Frontier sampling keeps LLM prompts more diverse than raw rollout host frequencies by reweighting which hosts receive prompt attention. The figure also reports CC2 checkpoint rewards by attacker type and method.

C. Mechanistic Analysis of Frontier Sampling

  • Sampled prompts remain more diverse than raw rollout streams, with host emphasis shifting as the active frontier moves.Early prompts deemphasize heavily represented hosts such as Op_Server0 and Enterprise2; later emphasis shifts with the frontier.Source: C. Mechanistic Analysis of Frontier… · p. 5, Figure 4 · p. 5
  • Figure 3 plots mean checkpoint evaluation reward for PPO, Ask the Expert, and PBRS variants in CC2, separately for B_lineAgent and RedMeanderAgent.Shaded bands denote ±1 standard error across independent training seeds.Source: Figure 3 · p. 5
  • Figure 4 compares normalized host-frequency entropy in rollout streams and prompts, then shows host-wise prompt-share minus rollout-share reweighting.Positive reweighting means a host receives more prompt emphasis than its rollout share.Source: Figure 4 · p. 5

VI. DISCUSSION

Across CC1 and CC2, Ask the Expert outperforms evaluated PBRS variants at every reported checkpoint for both attacker types and improves over PPO during early and intermediate training. It retains the strongest terminal mean in all four settings while deployment remains policy-based.

VI. DISCUSSION

  • Across CC1 and CC2, Ask the Expert exceeds every evaluated PBRS variant at each reported checkpoint for both attacker types and retains the strongest terminal mean in all four settings.Its improvement over PPO is substantial during early and intermediate training; the PBRS gap narrows in CC2 late in training.Source: §VI · p. 6
  • The frontier-sampling diagnostic shows prompts remain more diverse than raw rollout streams instead of collapsing onto the most frequent host.This diversity benefit persists throughout training.Source: §VI · p. 6
  • The results position Ask the Expert as an asymmetric training approach that uses semantic reasoning during training while keeping deployment lightweight and policy-based.Source: §VI · p. 6

VII. CONCLUSION

The conclusion presents Ask the Expert as training-time LLM-guided reward shaping that distills constrained host-level advice into tiered bonuses, then leaves deployment to a pure RL policy. On CAGE CC1 and CC2 with two attacker types, it improves sample efficiency over PPO, beats evaluated PBRS variants throughout reported learning curves, and retains the strongest terminal mean in all four settings.

VII. CONCLUSION

  • Ask the Expert converts defender-visible compromise signals into summaries, intermittently queries an LLM, and distills constrained host-level advice into tiered reward bonuses.The LLM is discarded after training, leaving a pure RL deployment policy.Source: §VII · p. 6
  • On CAGE CC1 and CC2 under two attacker types, the method improves sample efficiency over PPO and exceeds every evaluated PBRS variant throughout reported learning curves.Although the gap to PBRS narrows during late training in some settings, Ask the Expert retains the strongest terminal mean in all four settings.Source: §VII · p. 6
Loading 2610.09337v1…