PaperlayerView exact revision on arXiv

Source-linked AI summary

ACE: An AI-Orchestrated Intrusion Detection and Compliance Mapping Engine

Abdoulie Ceesay, Rishikesh Sahay, Md Rasel Al Mamun, Bell Raj Eapen

arXiv:2610.22702v1cs.CR

TL;DR

ACE addresses the gap between intrusion detection and compliance documentation by integrating alert interpretation, control mapping, and defensive rule deployment. In a laboratory evaluation, it processed all eight attack alerts, produced no false positives in three legitimate-traffic tests, and achieved an average NIST mapping confidence score of 0.87.

  • Problem

    IDS telemetry and compliance documentation are often handled separately, leaving a gap between detected events and control-level evidence.

    Source: §1 · p. 1, §3 · p. 5

  • Method

    ACE combines Suricata alert processing, LangGraph orchestration, RAG-based NIST control retrieval and ranking, and automated Suricata rule deployment.

    Source: §1 · p. 2, §9 · p. 16

  • Results

    ACE processed all eight attack alerts, produced no false positives across three legitimate-traffic tests, generated deployable rules for 37.5% of alerts, verified all generated-rule deployments, and achieved an average NIST mapping confidence score of 0.87.

    Source: §9 · p. 16

  • Takeaways & Limitations

    ACE demonstrates a repeatable evidence chain linking alerts, interpreted threats, control mappings, generated rules, and deployment verification.

    Source: §7 · p. 14

    The evaluation used a controlled two-VM environment and only eight attack scenarios, limiting evidence about enterprise-scale traffic and broader intrusion behavior.

    Source: §8 · p. 15

Abstract

from arXiv · show

This study proposes the Automated Compliance Engine (ACE), an AI-orchestrated framework that links intrusion detection, regulatory control mapping, and automated countermeasure deployment for security operations environments. ACE integrates Suricata IDS, a LangGraph five-stage workflow, retrieval-augmented generation, NIST SP 800-53 Rev. 5 control retrieval, large language model-based control ranking, and SSH-based Suricata rule deployment. The framework was evaluated in a two-virtual-machine laboratory using eight simulated attack scenarios and three legitimate traffic tests. ACE processed all eight attack alerts, achieved 100% alert detection, 0% false positives across legitimate traffic tests, generated deployable AUTO rules for 37.5% of alerts, verified 100% deployment of generated rules, and achieved an average NIST control-mapping confidence score of 0.87. The DeepEval assessment produced 1.0 scores for answer relevancy and faithfulness, while the hallucination score reflected known LLM-as-judge calibration limitations. The evaluation was conducted in a controlled environment with a limited attack set. Further validation is required using enterprise-scale telemetry, multiple IDS sensors, and an analyst-in-the-loop review. The results demonstrate how AI-assisted security automation can reduce analyst effort in translating IDS alerts into compliance-relevant evidence and defensive actions. ACE contributes an integrated detection-to-compliance-to-response workflow combining IDS telemetry, RAG-based control mapping, and automated rule deployment.

1. Introduction

ACE addresses the gap between IDS detection and compliance documentation by integrating alert interpretation, control mapping, and countermeasure deployment in one workflow. It combines retrieval-grounded control ranking with constrained rule deployment to connect monitoring, governance, and defensive action.

1. Introduction

  • Security operations must rapidly triage high-volume telemetry while documenting how detected events affect regulatory and control obligations.The paper describes alert detection and compliance documentation as often separate functions.Source: §1 · pp. 1–2
  • ACE integrates threat detection, compliance mapping, and countermeasure deployment in one auditable workflow using Suricata, LangGraph, RAG, NIST SP 800-53 Rev. 5, and SSH-based rule deployment.The system was implemented with a Kali Linux orchestrator and a Windows IDS host.Source: §1 · p. 2
  • ACE treats compliance mapping as part of detection because isolated IDS evidence otherwise leaves governance teams to reconstruct its control relevance after an incident.The workflow associates security events with relevant controls and a documented response path.Source: §1 · p. 2
  • ACE can generate narrowly scoped Suricata rules from actionable alert indicators, joining monitoring, governance, and defensive engineering in a feedback loop.The paper frames this as active security-control reinforcement rather than passive compliance reporting.Source: §1 · p. 2
  • The workflow grounds control mapping in a NIST vector database, routes execution through a deterministic supervisor, and validates rule generation through a structured registry.These constraints limit model autonomy in response to concerns about hallucination, non-determinism, and unsafe autonomous action.Source: §1 · p. 2
  • The study evaluates ACE with eight simulated attack scenarios and three legitimate-traffic tests, measuring detection, false positives, deployment, control-mapping confidence, and LLM output quality.The introduction presents this evaluation as the study’s third contribution.Source: §1 · pp. 2–3

2. Related Work

Prior work supplies components for IDS telemetry, retrieval-grounded compliance reasoning, explainability, and agentic security workflows, but these are often studied separately. ACE combines them in a defensive pipeline that connects live alerts to compliance mapping and rule deployment.

2. Related Work

  • Suricata produces structured event records useful for automated workflows, but it does not interpret an alert’s compliance meaning or create audit-ready control mappings.SIEM systems can correlate and enrich events, yet analysts often still connect technical events to governance frameworks.Source: §2 · p. 3
  • Prior work identifies risks in LLM security outputs and agentic systems, while ACE uses deterministic routing, constrained rule templates, and registry checks around model components.ACE’s architecture can also place human approval between rule generation and deployment.Source: §2 · pp. 4–5
  • Commercial playbooks and compliance dashboards offer partial automation, but are often manually authored or rely on asset and configuration evidence rather than semantic mapping from live alerts.ACE is presented as an AI-guided component that could integrate with, rather than replace, SOAR or SIEM platforms.Source: §2 · p. 4
  • Security systems need traceable explanations, and ACE preserves retrieved control text alongside rankings within structured workflow transitions.ACE’s explanation links the alert, threat interpretation, retrieved controls, and rule templates to their respective evidence or action.Source: §2 · p. 4
  • Because NIST controls use broad language that differs from IDS signatures, keyword search can miss semantic links; RAG retrieves candidate controls for contextual ranking.RAG-based compliance work includes retrieval grounding, multi-agent verification, and graph-enhanced retrieval [3].Source: §2 · pp. 3, 5
  • Existing IDS tools detect alerts but do not map them to compliance controls, while compliance-mapping and agentic systems often lack live-alert-to-response integration.ACE combines these functions and evaluates their interaction in a live laboratory environment.Source: §2 · p. 5

3. Research gap and contribution framing

The paper identifies the missing link as an integrated, auditable workflow from intrusion detection through compliance mapping to defensive rule deployment. ACE is framed as a controlled proof of concept, evaluated in a two-VM laboratory rather than as a production-ready system.

3. Research gap and contribution framing

  • Existing approaches lack a single workflow combining live alert detection, compliance mapping, rule generation, automated deployment, and cross-run deduplication.The paper identifies this integration as the central research gap.Source: §3 · p. 5
  • ACE automates repetitive post-alert translation steps—interpretation, control retrieval and ranking, countermeasure proposal, and audit-trail preservation—without aiming to replace analysts or governance processes.Its design objective is to reduce manual friction while maintaining determinism where operational safety requires it.Source: §3 · p. 6
  • ACE’s design principles are grounded autonomy, operational idempotence, and auditability, with probabilistic interpretation separated from deterministic rule generation and deployment.Deterministic templates and registry checks prevent free-form model text from becoming executable policy without validation.Source: §3 · p. 6
  • The evaluation asks whether ACE can ingest Suricata alerts and map controls, generate and deploy countermeasure rules, and assess limitations of LLM and RAG components.These questions address operational feasibility and governance implications of AI-enabled security automation.Source: §3 · pp. 6–7
  • The study evaluates a controlled two-VM implementation with real Suricata alerts, generated rules, and deployment verification as a proof-of-concept feasibility demonstration.The authors caution that this is not evidence that ACE is production-ready.Source: §3 · p. 7

4. ACE system architecture

ACE uses a two-VM architecture in which a Kali orchestrator processes Suricata alerts from a Windows IDS host and deploys proposed rules over SSH. Its workflow combines interpretation and retrieval with deterministic routing, constrained rule generation, and registry-based governance.

4. ACE system architecture

  • The pipeline checks an alert’s SID against a registry, interprets it, retrieves and ranks NIST controls, generates constrained Suricata rules, and deploys them to the IDS host over SSH.Alerts reach the Kali orchestrator through a shared folder from Suricata on the Windows host.Source: §4 · p. 7
  • The Kali Linux VM hosts orchestration, the LLM client, RAG, and attack simulation; the Windows VM hosts Suricata and receives deployed rules.The VMs communicate over a host-only network, using SSH for deployment and shared folders for log and proposal-file exchange.Source: §4 · p. 7
  • The system separates sensing, interpretation and retrieval, action, and governance into four functional layers.Suricata generates events; LangGraph, ChromaDB, and an embedding model support interpretation and retrieval; rule generation and deployment form the action layer; the Supervisor and registry support governance.Source: §4 · p. 7
  • A deterministic Supervisor routes the five-node StateGraph by checking state fields rather than making decisions from free-form model output.This makes the execution path predictable and reduces the possibility of decision-cascade errors.Source: §4 · p. 7
  • Interpret creates a structured threat description from alert fields, Retrieve returns the top three NIST controls, and Rank scores their relevance from 0 to 1.Generate extracts indicators from the original alert in a priority order, while Deploy writes proposed rules and transfers them over SSH.Source: §4 · p. 8
  • SID checks skip previously processed alerts before any LLM calls, and rule fingerprinting prevents repeat generation of equivalent rules.Deployment security guidance calls for least-privilege accounts, key rotation, command restrictions, logging, and rollback in enterprise use.Source: §4 · p. 8
  • ACE retains the original alert in workflow state so indicator extraction can return to authoritative metadata instead of relying on an LLM paraphrase.The threat description supports retrieval and ranking, while rule generation uses the original Suricata event.Source: §4 · p. 9

5. Experimental Design

ACE was evaluated in a controlled laboratory experiment spanning simulated attacks and legitimate traffic. The design assessed detection, control mapping, rule generation and deployment, false-positive behavior, and LLM output quality, with traceable processing and deterministic checks.

5. Experimental Design

  • Eight simulated attacks across four categories and three legitimate traffic tests formed the controlled evaluation set.The attack categories were reconnaissance, web application attacks, DNS abuse, and denial-of-service activity; scenario metadata varied in suitability for rule generation.Source: §5 · pp. 9–10
  • NIST controls were embedded into a ChromaDB vector index, and the three most similar controls for each threat description were ranked using retrieved text.The offline embeddings used all-MiniLM-L6-v2 and an HNSW index; the online workflow ranked retrieved controls rather than relying solely on model knowledge.Source: §5 · p. 10
  • Deployment testing checked whether generated AUTO rules appeared in the Windows host’s signatures.rules file after deployment.Rules were written to proposed_rules.rules first, making the check an end-to-end test of defensive action rather than local proposal generation alone.Source: §5 · p. 10
  • False-positive testing compared alert-signature snapshots before and after three benign activities, counting only newly appearing signatures.The activities were a normal HTTP GET request, wget traffic, and a normal DNS lookup; the snapshot approach preserved prior attack evidence.Source: §5 · p. 10
  • Mapping quality considered Rank-node confidence and whether retrieved controls were plausible for the alert and control family, not exact legal conclusions.Mappings were treated as candidate compliance associations that could accelerate analyst review.Source: §5 · p. 11
  • Evaluation combined five phases with deterministic SID and artifact checks, while treating DeepEval relevancy, faithfulness, and hallucination scores as supporting evidence.The methodology also used fixed node sequencing and a persistent registry of processed SIDs and generated rules for traceability.Source: §5 · pp. 11–12

6. Results

ACE processed all eight attack alerts and mapped them to NIST controls, while rule generation depended on whether alerts contained suitable indicators. All generated rules were deployed, and the evaluation also reported no false positives in the tested benign traffic and high average mapping confidence.

6. Results

  • 100% of the eight attack alerts were processed and mapped to NIST controls, while rule generation remained conditional on suitable indicators.The result does not mean every alert produced a deployable countermeasure rule.Source: §6 · pp. 12–13
  • 37.5% of alerts produced deployable indicator-specific AUTO rules, corresponding to three of eight alerts.The remaining alerts lacked suitable extractable metadata or did not satisfy extraction constraints.Source: §6 · p. 12
  • 100% of generated rules were deployed successfully, with all six rules verified on the Windows IDS host.Deployment was constrained through deterministic templates.Source: §6 · p. 12
  • 0% false positives were measured across the three legitimate traffic tests, with no new alerts during those activities.The paper cautions that the benign test set was small.Source: §6 · p. 12
  • 0.87 was the average NIST control-mapping confidence score across eight alerts and three retrieved controls per alert.The paper describes the result as supporting semantic alignment between interpreted threats and retrieved controls.Source: §6 · p. 12
  • DeepEval scores were 1.0 for answer relevancy and 1.0 for faithfulness; the hallucination score was 0.67 across two of three cases.The authors caution that LLM-as-a-judge metrics can diverge and rely primarily on deterministic operational checks for validation.Source: §6 · pp. 12–13
  • The eight-alert pipeline run required approximately 173 seconds in the laboratory environment.The reported runtime includes LLM calls, retrieval, rule generation, and deployment overhead.Source: §6 · p. 13
  • HTTP fields supported the richest rule generation, while several scan and network-activity alerts mapped to controls without indicator-specific rules.The SYN scan produced a source-IP fallback rule, which the paper says requires care in production because source IPs can be spoofed, shared, or dynamic.Source: §6 · p. 13

7. Discussion

ACE links alert interpretation, control mapping, and response evidence in one workflow, while constraining rule generation and preserving human review for high-impact decisions. The discussion presents this integration as a way to make compliance evidence more operational and security response more accountable.

7. Discussion

  • The 0.87 average confidence score suggests control ranking can produce useful candidate mappings for analyst review.ACE combines IDS metadata, LLM-based interpretation, and retrieval-grounded control mapping to partially automate alert translation.Source: §7 · p. 14
  • ACE’s repeatable chain links raw alerts, interpreted threats, retrieved and ranked controls, generated rules, and deployment verification, creating evidence during detection for audit readiness.The workflow records which controls were retrieved, the assigned confidence scores, and whether a countermeasure was deployed.Source: §7 · pp. 14–15
  • Only three of eight alerts yielded rules; ACE distinguishes documentation-only, rule-generating, and human-escalation outcomes rather than changing controls for every alert.For FIN scans, XMAS scans, and UDP floods, the current stronger value is compliance mapping and documentation rather than indicator-specific rule generation.Source: §7 · pp. 14–15
  • ACE is a decision-support and evidence-generation system, not a final authority; human review should remain for high-impact deployments and formal audit responses.Review is especially important when rules could affect business-critical traffic or mappings are used in formal audit responses.Source: §7 · p. 14
  • Integrating compliance context into detection and response workflows can make evidence more operational and security response more accountable.ACE aligns low-level telemetry with a structured control catalog and carries the mapping into response artifacts.Source: §7 · pp. 14–15
  • Deterministic routing, rule templates, deployment verification, and SID-registry deduplication form layered safeguards; the registry supports idempotent runs and records processed SIDs and generated rules.ACE keeps routing, deduplication, rule construction, and deployment logic deterministic, using the LLM for semantic interpretation.Source: §7 · pp. 14–15

8. Limitations and future work

The evaluation and validity claims are bounded by a controlled, small-scale laboratory, limited scenarios, and confidence scores used as a proxy for mapping quality. Future work calls for broader comparisons, validation, safeguards, and support for additional compliance frameworks.

8. Limitations and future work

  • Evaluation in a two-VM lab with eight attack scenarios and three legitimate-traffic tests limits generalizability to enterprise traffic, other sensors, and false-positive behavior.The study did not test multiple network segments or heterogeneous IDS sensors, and the attack set does not represent the full range of modern intrusion behavior.Source: §8 · p. 15
  • Confidence scores serve as a proxy for mapping quality, seed rules may not represent production detection logic, and independent replication would strengthen findings.The authors recommend expert analyst ratings to assess mapping quality.Source: §8 · p. 15
  • Comparisons with manual mapping, keyword-only retrieval, SIEM-only enrichment, and RAG without LLM ranking, plus ablations, could clarify component contributions.Suggested ablations remove the registry, ranking step, or RAG layer.Source: §8 · pp. 15–16
  • Rule generation requires extractable indicators, such as HTTP, TLS, DNS, or payload fields, limiting coverage for transport-layer and volumetric events.The authors identify this as a deliberate safety constraint.Source: §8 · p. 16
  • Single-stage RAG retrieval may benefit from re-ranking, graph-based retrieval, or control-family constraints, while DeepEval is not a complete measure of output correctness.The authors describe DeepEval as providing useful signals that should not be treated as a complete quality measure.Source: §8 · p. 16
  • Future validation should use larger datasets, additional IDS tools, realistic enterprise traffic, analyst review, and production safeguards such as rollback and access controls.The authors also propose SIEM case-management integration and comparisons across LLMs, embedding models, and RAG strategies.Source: §8 · p. 16
  • Extending ACE beyond NIST SP 800-53 Rev. 5 to frameworks such as ISO/IEC 27001, CIS Controls, and PCI DSS could support evidence across compliance regimes.The proposed multi-framework mapping layer would allow one alert to generate evidence across several regimes.Source: §8 · p. 16

9. Conclusion

ACE integrates intrusion detection, NIST control mapping, and Suricata rule deployment in an AI-orchestrated workflow. In a laboratory evaluation, it processed all eight attack alerts, produced no false positives across three legitimate traffic tests, and verified deployment of all generated rules, while the authors frame it as support for—not replacement of—security professionals.

9. Conclusion

  • ACE combines Suricata IDS, LangGraph orchestration, RAG retrieval, LLM-assisted control ranking, deterministic rule generation, SSH deployment, and SID-based deduplication.Source: §9 · p. 16
  • ACE processed all eight attack alerts, produced no false positives across three legitimate traffic tests, generated deployable rules for 37.5% of alerts, verified deployment of all generated rules, and achieved an average NIST mapping confidence score of 0.87.Source: §9 · p. 16
  • ACE automates repetitive translation and documentation while preserving auditable evidence and deterministic safeguards; it is not a replacement for SOC analysts or governance professionals.The authors identify enterprise-scale testing, human-in-the-loop controls, and broader framework support as needed for further development.Source: §9 · p. 16
Loading 2610.22702v1…