Source-linked AI summary
Verification-Guided Specification Synthesis with Large Language Models for Intrusion Detection Rules
Kohei Yamamoto, Marie Katsurai
TL;DR
Observed IoT attack traffic must be converted into deployable IDS rules despite manual authoring burdens and limited immediate auxiliary information. The paper synthesizes and verifies semantic specifications from HTTP traces with CEGIS before deterministic Suricata compilation, achieving 81.5% detection with 0.0% false positives.
Problem
Converting observed IoT attack traffic into deployable IDS rules remains manual, while existing approaches may require unavailable auxiliary information or omit benign-traffic validation.
Method
An LLM identifies a vulnerable parameter and synthesizes semantic detection specifications from HTTP traces, which CEGIS refines using benign counterexamples before deterministic Suricata compilation.
Results
81.5% detection rate with 0.0% false-positive rate was achieved on 281 real-world CVEs and benign IoT communications.
Takeaways & Limitations
Counterexample-guided verification improves detection performance, while deterministic compilation supports operationally consistent rule construction.
Takeaways & Limitations
HTTP-request-only analysis makes attacks depending primarily on server-side state or response content difficult to characterize.
Abstract
from arXiv · showhide
Attacks against Internet-connected IoT devices continue to increase; however, transforming observed attack traffic into deployable intrusion detection system (IDS) rules remains largely a manual process. Recent studies have explored using large language models (LLMs) to generate IDS rules; nonetheless, existing approaches often require auxiliary information beyond observed traffic or generate rules without validating their detection logic against benign traffic. This study presents a verification-guided specification synthesis framework for generating Suricata rules directly from HTTP request traces. Instead of having an LLM generate IDS rules in a single step, an LLM first identifies a vulnerable parameter and synthesizes a semantic detection specification. These specifications are iteratively refined through counterexample-guided inductive synthesis (CEGIS), in which benign traffic samples serve as counterexamples during synthesis and verification. Verified specifications are then deterministically compiled into Suricata rules. Experiments on 281 real-world CVEs and benign traffic collected from real IoT devices show that the proposed method achieves a detection rate of 81.5% while maintaining a false positive rate of 0.0%. An ablation study also demonstrates that CEGIS-based verification improves detection performance while maintaining a low false positive rate.
1 Introduction
The paper addresses the manual, expertise-intensive conversion of observed IoT attack traffic into deployable IDS rules. It proposes verification-guided LLM specification synthesis directly from HTTP request traces, followed by deterministic Suricata compilation.
- IoT devices often lack firmware updates, making network-based IDS protection important against continuing large-scale attacks.
- Transforming newly observed attack traffic into deployable IDS rules remains manual, expertise-intensive, and delay-inducing.
- Existing LLM-based approaches often require auxiliary sources and generate Suricata rules without validating detection logic against benign traffic.
- The proposed framework has the LLM identify vulnerable parameters and synthesize semantic specifications, which CEGIS refines before Suricata compilation.
2 Related Work
Prior IDS signature methods use statistical or auxiliary attack information, while recent LLM approaches commonly generate rules in one step without benign counterexamples. The proposed method instead operates directly on HTTP request traces and verifies semantic specifications before compilation.
- Statistical signature methods can miss attacks whose syntax closely resembles legitimate communications.
- Recent LLM-based methods use sources such as CTI reports, exploit code, or HTTP attack requests to generate detection rules.
- These LLM-based approaches generally generate detection rules in a single step without using benign traffic as construction-time counterexamples.
- The proposed method requires neither CTI reports, exploit code, attacker labels, nor labeled benign corpora, operating directly on HTTP request traces.
- It synthesizes semantic detection specifications and refines them with benign counterexamples through CEGIS before compiling Suricata rules.
3 Proposed Method
The method separates semantic specification synthesis and verification from deterministic Suricata rule compilation. An LLM identifies a vulnerable parameter, proposes a specification using attack and benign examples, and iteratively refines it through CEGIS verification.
- The framework operates directly on HTTP request traces and separates specification synthesis and verification from rule compilation.This design avoids requiring HTTP responses, firmware images, or vulnerability descriptions.
- The LLM selects a likely vulnerable parameter according to its operational role, such as command execution, data access, path construction, or access control.
- The LLM synthesizes a semantic detection specification from the attack value and benign parameter values rather than generating IDS syntax directly.
- 3.1 Verification-Guided Specification Synthesis: CEGIS iteratively refines candidate specifications using verifier-produced counterexamples and a mock server emulating the target service.Verification checks attack responses, parameter influence on behavior, benign-value exclusion, and parameter observability in logs.
- 3.2 Rule Compilation: Verified specifications are deterministically compiled into Suricata rules using request characteristics and the identified parameter.
4 Experiments
Experiments evaluate the framework on 281 CVEs with realistic benign IoT traffic, comparing detection and false-positive performance across methods and ablations. The proposed method achieves 81.5% DR at 0.0% FPR, while CEGIS refinement substantially improves detection and failures concentrate in attacks lacking request-level discriminators.
- Experimental Setup: 281 CVEs and 281 benign IoT communications were used to evaluate generated rules across three random seeds and identical attack traces, settings, and LLM configurations.The CVEs included IoT or network-device vulnerabilities exploitable through a single HTTP request; benign traffic came from real IoT devices.
- Evaluation Metrics: Detection rate (DR) and false positive rate (FPR) were measured at Suricata’s triggering layer, with DR assessing rule-generation success on the source attack traces.FPR represents benign IoT communications that incorrectly trigger generated rules.
- Detection and False-Positive Performance: 81.5% DR and 0.0% FPR were achieved by the proposed method, making it the only evaluated approach with high detection and zero false positives simultaneously.The remaining 18.5% of attack instances were cases where no Suricata rule could be generated.
- Detection and False-Positive Performance: Deterministic rule construction supports the low FPR by restricting matches to specific endpoints, vulnerable parameters, and attack-derived conditions.These contextual restrictions make unrelated benign traffic unlikely to satisfy the generated matching conditions.
- Ablation Study: Removing CEGIS refinement reduced DR from 81.5% to 41.3%, while removing counterexample diagnosis reduced DR to 57.8%; FPR remained 0.0% in both ablations.Removing the compiler reduced DR by 10.7 percentage points, and none of the ablations increased deployment-time FPR.
- Failure Analysis: The proposed method’s failures primarily occur when CEGIS cannot synthesize a specification matching the attack value while rejecting all benign values.Command-injection and path-traversal attacks are usually detected, whereas attacks depending on server-side state or response content remain difficult to characterize from HTTP requests alone.
5 Conclusion and Future Work
The paper’s CEGIS-based framework generates IDS rules from HTTP request traces and was evaluated on real-world CVEs and benign IoT communications. Future work targets generalization, server-side attack classes, and mixed-traffic deployment.
- 81.5% detection rate and 0.0% FPR were achieved across 281 real-world CVEs and 281 benign IoT communications.The ablation study identified counterexample-guided verification as the primary contributor to detection performance.
- Future benchmarks should test multiple attack variants per vulnerability to evaluate generalization to unseen payloads and encodings.
- Extending verification with real devices or emulation may support attack classes whose maliciousness depends mainly on server-side state.The passage names authentication bypass and information disclosure vulnerabilities as examples.
- Practical deployment requires evaluating robustness when attack and benign traffic are mixed rather than separated at collection points.The current framework assumes attack traffic collected from honeypots or monitoring points.
6 GenAI Usage Disclosure
The authors disclose using ChatGPT and Claude for manuscript editing, translation, and code development, with outputs verified and modified by the authors.
- ChatGPT also generated decorative icons for Figure 1, used only for visualization.