PaperlayerView exact revision on arXiv

Source-linked AI summary

SENTINEL-RL: Offloading Topological Reasoning from LLM Agents in the Security Operations Center

Uday Vallabhaneni, Cassie L. Cagwin, David J. Wild

arXiv:2609.04159v1cs.CRcs.AI

TL;DR

Enterprise SOC agents struggle to represent large authentication graphs and to guarantee topology-consistent containment through free-form generation. Sentinel-RL separates graph-based topological reasoning from LLM narration using HetGAT, PPO, constrained actions, and critic gating, and reports successful ingestion, alerting, policy, and end-to-end evaluations.

  • Problem

    LLM SOC agents cannot fit multi-thousand-host authentication graphs into context windows and cannot guarantee that free-form containment recommendations respect network topology.

  • Method

    Sentinel-RL uses a HetGAT encoder and PPO policy for topology-aware constrained decisions, while an LLM produces narratives gated by a critic.

  • Results

    The system ingests a 24-million-edge subgraph in 14.2 minutes, alerts within 2.45 seconds at the 99th percentile, reaches 0.91 precision and 0.87 recall, and completes the lifecycle in 6.3 seconds median.

  • Takeaways & Limitations

    Two-phase CREATE ingestion, anchor-node co-location, and enterprise control mechanisms are presented as transferable patterns for institutional deployment.

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly proposed as autonomous SOC analysts, but two limitations make them unreliable at enterprise scale: a finite context window cannot hold a multi-thousand-host authentication graph, and free-form generation offers no guarantee that a recommended containment action is consistent with the topology it operates on. We present Sentinel-RL, an agentic-SOC architecture that decouples topological reasoning from semantic reasoning: a heterogeneous graph attention encoder summarizes the live authentication subgraph into a fixed-dimensional state, a Proximal Policy Optimization (PPO) policy maps this state to a constrained set of investigative actions, and an LLM agent loop is restricted to consuming the policy's recommendations and producing analyst-readable narratives gated by a critic. We instantiate the system on the LANL Comprehensive, Multi-Source Cyber-Security Events dataset and the Indiana University Quartz HPC cluster, reporting four results: (i) a two-phase CREATE ingestion pattern loads a 24M-edge authentication subgraph into Neo4j in 14.2 minutes on a single 32-core node, roughly 24x faster than the canonical MERGE-based pipeline; (ii) a sliding-window alert engine reliably trips a 25-event/10-second threshold in <=2.5 s across 50 trials; (iii) PPO training over 200 iterations converges to a mean episodic return of 8.74+/-0.31, with held-out precision of 0.91 and recall of 0.87 on labeled red-team events; and (iv) the integrated containment loop completes a full detect-investigate-recommend-human-approve cycle in a median of 6.3 s. We contribute a reusable engineering pattern (the hot-node deadlock workaround), a portable HPC deployment pattern (anchor-node co-location), and an enterprise-readiness analysis covering false-positive economics, reversibility guarantees, audit compliance, and the human-approval boundary.

I. Introduction

Sentinel-RL addresses two enterprise-scale SOC limitations: LLMs cannot represent large authentication topologies in context, and free-form generation cannot guarantee topology-consistent containment. It assigns topology-aware decisions to a trained policy while retaining the LLM for semantic interaction and adds reusable deployment patterns and enterprise controls.

  • Motivation: 10^4–10^5 host nodes and 10^6+ recurring connections can exceed LLM context windows, making lateral-movement reasoning unreliable.The paper frames topology compression as necessary for tracking attacker pivots and evaluating whether isolation severs access.
  • Motivation: Containment actions such as isolate-host, block-ip, and disable-account require hard constraints, auditability, and version-consistent execution.The paper argues that probabilistic text completion provides none of these guarantees.
  • Approach: Sentinel-RL assigns topology-aware network decisions to a trained policy while restricting the LLM to queries, alert summaries, and reports.This division of labor is presented as a hybrid neuro-symbolic architecture.
  • Approach: A four-layer architecture separates Neo4j graph data, a HetGAT-plus-PPO decision engine, telemetry ingestion, and the user interface.The architecture was tested against red-team events in the LANL benchmark.
  • Engineering contributions: The implementation contributes a two-phase CREATE ingestion workaround and an HPC strategy for co-locating microservices on SLURM-managed nodes.These patterns address parallel-ingestion deadlocks and deployment placement.
  • Evaluation: The evaluation measures ingestion throughput, pipeline latency, policy convergence, detect-to-contain latency, false-positive economics, reversibility, audit compliance, and human approval.The paper treats deployment controls and approval boundaries as part of enterprise readiness.

A. The reality of lateral movement

Sentinel-RL treats lateral movement as a graph-routing problem over authentication relationships rather than as isolated host events. Its graph-based state representation and PPO policy support topology-aware investigative actions under explicit threat-model and trust assumptions.

  • Threat context: Lateral movement turns incident response into a graph-routing problem because attackers pivot across systems using hijacked or elevated credentials.The operational question is which machine to quarantine within the surrounding authentication topology.
  • Threat context: The LANL dataset spans 58 days, about 1.65×10^9 events, 12,425 users, and 17,684 endpoints with verified red-team activity.It combines anonymized Windows authentications, DNS lookups, and NetFlow records.
  • Prior evidence: Graph-based detection models are reported to outperform tabular or sequential-log approaches, with LANL evaluations showing 85%–99% true-positive rates and false positives under 5%.Related autonomous-defense studies also report better adaptation to new network layouts with heterogeneous graph encoders than with standard MLPs.
  • State representation: HetGAT compresses ongoing authentication activity into a dense 64-dimensional vector for the policy engine.The representation preserves relational structure rather than flattening the live network into serialized text.
  • Policy choice: PPO is selected for clipped policy updates, reproducible training, Ray RLlib integration, and regression testing on held-out scenarios.These properties are presented as useful for predictable and auditable discrete containment decisions.
  • Threat model: The threat model begins after perimeter compromise and focuses on lateral movement through valid accounts and remote services.The defender continuously ingests authentication telemetry but initially does not know which assets are compromised.
  • Trust assumptions: Security guarantees rely on an internal critic, policy auditor, human approval, and hard action masking rather than assuming perfect RL decisions.The architecture explicitly accounts for inevitable LLM hallucination risk.

IV. Architecture

Sentinel-RL uses modular planes to separate live graph memory, topology-aware policy decisions, telemetry triggers, and analyst interaction. The strategic plane encodes a local authentication neighborhood and selects from five constrained investigative actions.

  • Architecture: The four-tier architecture isolates operational responsibilities so components can be swapped, audited, or upgraded independently.Modularity is intended to avoid disrupting the broader pipeline.
  • Data plane: The data plane stores timestamped host, user, service, segment, authentication, and connection relationships in Neo4j.Ray Data shards LANL ingestion by source host and exposes streaming-ingest and Cypher interfaces.
  • Strategic plane: A two-hop neighborhood around suspicious activity is encoded by HetGAT as s_t ∈ R^64 before PPO selects an investigative action.The encoder produces a dense state from the localized topological map.
  • Strategic plane: PPO selects one of five actions: QueryEDR, QueryAD, CheckThreatIntel, ExamineFirewall, or TerminateAndOutputVerdict.The action set constrains policy outputs to predefined investigative operations.
  • Deployment boundary: Serving the policy through FastAPI creates version control, shadow deployment, and a governance interception boundary.External tools can scrutinize proposed actions before execution.
  • Telemetry plane: A sliding-window trigger fires when a source host initiates N=25 authentications within W=10 s.The webhook passes the host identifier and qualifying time window to orchestration.

D. Orchestration plane

The orchestration and deployment design connects policy outputs to analyst oversight while addressing graph-ingestion contention and HPC placement. Its control surface combines LLM narration, critic review, auditable constraints, and a two-phase database-loading pattern.

  • Orchestration plane: A Triage agent converts PPO output into analyst-readable explanations while a Critic agent checks whether sufficient evidence supports the proposed action.The orchestration plane uses Streamlit, PyVis, and LangChain to mediate human interaction.
  • Safety surface: Action masking blocks TerminateAndOutputVerdict until evidence from at least two distinct sources is gathered.An immutable audit log records network state, action, timestamp, and policy version for every decision.
  • Deployment: All components are co-located on one HPC anchor node, keeping inter-plane communication on loopback while retaining local Ray and PyTorch parallelism.The allocation uses 32 CPU cores and 128 GB RAM.
  • Ingestion bottleneck: Per-row parallel MERGE ingestion collapsed on 24 million edges because workers contended for Neo4j locks on shared high-degree nodes.Distinct edges still serialized when their endpoints overlapped.
  • Two-phase CREATE pattern: The two-phase pattern pre-materializes vertices sequentially, then uses MATCH and explicit CREATE for parallel edge insertion.The edge phase avoids endpoint-lock contention after all structural vertices exist.
  • Engineering insight: The pattern preserves parallel edge generation by separating structural vertex creation from heavy edge insertion.The initial sequential pass is described as a small time cost that is outweighed by eliminating thread contention.

B. Consolidating services: the anchor-node pattern

Sentinel-RL co-locates latency-sensitive services on one anchor node because cross-node SLURM deployment caused silent connectivity failures and latency risks. Distributed execution is reserved for parallelizable workloads.

  • Failure mode: Cross-node service placement caused empty policy responses, lost webhooks, and an unreachable graph database.The failures arose when SLURM distributed Neo4j, FastAPI, alerting, and UI services across compute nodes.
  • Failure mode: Quartz lacks automatic routing for generic inter-node TCP traffic, while batch-job contention introduces severe communication latency spikes.Login nodes were unsuitable because of memory caps and outbound connectivity blocks.
  • Anchor-node solution: Co-locating all four architectural planes on a 32-core, 128 GB anchor node enables loopback communication and preserves local service connectivity.Ray processing and PyTorch evaluation remain parallelized within the anchor node.
  • Deployment rule: The anchor-node pattern reserves distributed execution exclusively for massive batch workloads where network latency will not disrupt application logic.Latency-sensitive interactive microservices remain consolidated on one heavy compute node.

C. Migrating RLlib hyperparameters

Sentinel-RL addresses an RLlib API transition by documenting the migration from the deprecated Trainer API to the RLModule/Learner structure used in its Ray 2.30+ environment. The evaluation runs on a dedicated 32-core Quartz anchor node without GPU acceleration.

  • API migration: Ray 2.30+ required migrating Sentinel-RL from the legacy Trainer API to the RLModule/Learner structure.The transition was described as underdocumented and disruptive to research workflows.
  • Configuration mapping: Table I provides a direct translation matrix for the critical hyperparameter configurations used during evaluation.The matrix documents the mapping associated with the framework migration.
  • Evaluation setting: All benchmarks ran on a Quartz anchor node with 32 Intel Xeon cores and 128 GB of RAM without GPU acceleration.The evaluation covered ingestion, alert responsiveness, PPO convergence and accuracy, and detect-to-contain latency.

A. Ingestion throughput

The evaluation spans ingestion, alert responsiveness, PPO learning, and held-out detection performance. The two-phase ingestion design avoids hot-node contention, while the alert engine responds within a few seconds and the policy converges stably.

  • Ingestion throughput: 852 seconds processes 24 million edges with the two-phase pipeline, versus 21,600 seconds estimated for single-phase MERGE.The optimized method remains near-linear as data volume grows, while MERGE degrades exponentially from compounded lock contention.
  • Ingestion throughput: The two-phase CREATE pipeline scales linearly, whereas single-phase MERGE scales super-linearly because of hot-node lock contention.Figure 2 presents the comparison on a log–log wall-time versus edge-count plot.
  • Alert responsiveness: 2.31 s mean webhook latency, 2.45 s 99th-percentile latency, and 2.48 s maximum delay were recorded across 50 alert-engine trials.Trials used a synthetic 10 event / s stream with thresholds of (W, N) = (10 s, 25).
  • Policy training: 8.74 ± 0.31 mean episodic return was reached after 200 PPO iterations across five independent seeds.The policy acquired most of its strategic capability within the first 80 rounds.
  • Detection performance: 0.91 precision and 0.87 recall were achieved on a sequestered eight-day LANL timeframe containing labeled red-team events.The held-out evaluation reports Sentinel-RL results across five training seeds.

D. End-to-end response latency

The integrated incident pipeline reaches a fully explained recommendation in just over six seconds, with algorithmic decision-making under 100 ms and natural-language generation accounting for most of the delay. The architecture places high-stakes irreversible actions behind human authority.

  • End-to-end latency: 6.33 s median is the complete end-to-end latency from initial warning to an explained recommendation.Table IV reports medians across 100 trial runs.
  • Latency breakdown: Under 100 ms is required for graph extraction, topology encoding, and PPO inference combined.The natural-language generation stage accounts for the bulk of total elapsed time.
  • Latency breakdown: 60% of total elapsed time is spent by the localized 8B, 4-bit quantized LLM drafting the analyst narrative.This makes narrative generation the dominant component of the response budget.
  • Autonomy boundary: High-stakes irreversible network modifications remain hard-gated so a human operator retains final authority.Safe read-only gathering is continuous, while reversible containment requires Critic Agent validation.

A. False-positive and false-negative economics

Sentinel-RL addresses both false-positive workload and false-negative risk through confidence filtering, human escalation, ensemble detection, regression testing, reversible-action controls, sandboxing, and auditable records.

  • False-positive economics: False alarms consume enterprise-scale labor, so the PPO policy filters predictions by historical frequencies and a strict confidence barrier.Low-confidence alerts are archived for optional human review, while investigative actions receive a reward penalty.
  • False-negative economics: False negatives trigger safeguards including entropy-based human intervention, a traditional statistical ensemble, and regression tests preserving held-out recall.An alert from either the policy or ensemble forces an investigation.
  • Reversibility: Autonomous operation is restricted to reversible countermeasures, while irreversible changes require explicit human authorization and a reversal script.The system categorizes actions by reversibility before permitting autonomous execution.
  • Pre-execution safety: A Response Sandbox tests proposed containment commands against a digital twin to identify cascading failures before execution.The sandbox is intended to prevent actions such as quarantining the only active DNS server or severing access to a critical authentication node.
  • Auditability: Every action, reasoning step, policy version, confidence metric, evidence item, state hash, timestamp, and human sign-off is preserved in a cryptographically secured ledger.The records are mapped to ISO 27001 and NIST 800-53 controls for regulatory review.

E. Human-approval boundary

Sentinel-RL places autonomous SOC behavior behind a tiered permission architecture and critic validation, while positioning graph-based reasoning and live-environment execution as alternatives to unconstrained LLM operation.

  • Permission boundary: Tier 1 intelligence gathering runs autonomously, Tier 2 reversible containment requires Critic Agent validation, and Tier 3 high-stakes actions require stronger authorization.The permission architecture assigns different approval requirements to querying platforms, quarantining inboxes, and higher-risk actions.
  • Operational scope: The system emphasizes bridging passive detection and automated containment rather than pursuing only incremental detector-accuracy gains.Its classifier is embedded in a hardened response architecture designed to shift from identifying threats toward safely neutralizing them.
  • Design basis: Sentinel-RL builds on graph-native perception and sparse objective-driven rewards for reinforcement-learning agents operating across networks.The cited prior work is presented as motivating graph perception for cross-network generalization and sparse rewards for learning effectiveness.
  • Deployment setting: Unlike work centered on CybORG and CybORG++, Sentinel-RL focuses on executing its modular architecture in live environments.The architecture is described as able to incorporate emerging acceleration techniques.

C. Agentic SOC architectures

Sentinel-RL proposes an agentic SOC architecture that separates graph-based topological reasoning from language-model narration and supports enterprise deployment through measured performance, transferable engineering patterns, and reproducible evaluation.

  • Evaluation rigor: The evaluation is designed to avoid spatial and temporal bias, laboratory-only testing, and unrealistic threat assumptions, with numerical results intended to be reproducible from documented artifacts.The reproducibility materials include Cypher queries, alert thresholds, PPO hyperparameters, and SLURM allocation details.
  • Architecture: The architecture offloads topological reasoning from the LLM to a dedicated graph encoder and PPO-driven policy, forming a scalable neuro-symbolic agentic SOC.The LLM is not the sole decision-maker for topology-aware containment.
  • Evaluation: The system ingests a 24-million-edge authentication subgraph in 14.2 minutes.This result is reported as part of the LANL end-to-end evaluation.
  • Evaluation: The alert pipeline triggers in under 2.45 seconds at the 99th percentile, while PPO reaches a mean episodic return of 8.74 within 200 iterations, with 0.91 precision and 0.87 recall on unseen data.These metrics cover alert latency, policy convergence, and held-out detection performance.
  • Evaluation: The full operational lifecycle from anomaly detection to synthesized containment recommendation completes in a median of 6.3 seconds.The reported lifecycle ends at the containment recommendation rather than an executed irreversible action.
  • Deployment contribution: The paper identifies two-phase CREATE ingestion, HPC anchor-node co-location, and enterprise controls as transferable patterns for deployment.These patterns are presented as helping move the system beyond a fragile research prototype.
Loading 2609.04159v1…