PaperlayerView exact revision on arXiv

Source-linked AI summary

Sigma-Hunter: A Domain-Specific Language Model for Threat Hunting and Detection Engineering

Kemal Davaslioglu, Sastry Kompella

arXiv:2610.09007v1cs.AIcs.CR

TL;DR

Threat information often omits the sources, fields, filters, and false-positive controls needed for operational rules, while general-purpose LLMs can produce invalid YAML or weak detection logic. Sigma-Hunter derives analyst-facing instruction examples from validated Sigma rules, partitions data by source rule before expansion, and fine-tunes models for rule generation and related tasks. Sigma-Hunter-Mistral achieved the highest syntax, detection-logic, completeness, selectivity, and overall scores, unlike untuned Mistral (4.61); syntax validity alone tracks semantic rule quality only loosely.

  • Problem

    Threat information often omits the sources, fields, filters, and false-positive controls needed for operational rules, while general-purpose LLMs can produce invalid YAML or weak detection logic.

    Source: §I · p. 1

  • Method

    Sigma-Hunter derives analyst-facing instruction examples from validated Sigma rules, partitions data by source rule before expansion, and fine-tunes models for rule generation and related tasks.

    Source: §I · p. 1, §III · p. 3

  • Results

    Sigma-Hunter-Mistral achieved the highest syntax, detection-logic, completeness, selectivity, and overall scores, unlike untuned Mistral (4.61); syntax validity alone tracks semantic rule quality only loosely.

    Source: §I · p. 2, §VI · p. 5

  • Takeaways & Limitations

    The adapted models support locally generated, reviewable detection drafts for disconnected environments, while generated rules remain subject to analyst validation before deployment.

    Source: §VII · p. 6

    Generated rules require local telemetry validation, false-positive tuning, and analyst approval before deployment.

    Source: §VII · p. 6

Abstract

from arXiv · show

Detection engineers must translate threat reports, forensic observations, and hunt hypotheses into precise, testable rules. General-purpose large language models (LLMs) can draft such rules, but often produce invalid YAML, incorrect log sources, unsupported fields, or overly broad detection logic. This paper presents \emph{Sigma-Hunter}, a domain-adapted LLM for analyst-assistive Sigma rule generation and threat hunting. We build an instruction-tuning dataset from 3,635 validated open-source Sigma rules, expanded into 7,663 question-answer and analyst-reasoning examples. Each source rule is assigned to a single train, validation, or test partition before this expansion, so no rule leaks across splits. We fine-tune a 7B Mistral model and a Phi-4 model with LoRA and score held-out rule generations on syntax, approximate field consistency, and a semantic judgment of detection logic, completeness, selectivity, and log-source alignment. Sigma-Hunter-Mistral scores 8.17 overall, against 7.88 for the strongest general-purpose baseline and 4.61 for untuned Mistral. Two findings stand out: domain adaptation enables a compact 7B model to perform competitively with larger general-purpose models on this structured task, and syntactic validity is a weak proxy for semantic rule quality, as several baselines emit well-formed YAML carrying weak detection logic. The adapted models run locally, which suits detection engineering in disconnected environments where analysts cannot reach hosted model services.

I. INTRODUCTION

Detection engineering requires turning threat information into testable detection logic, but general-purpose LLMs can produce plausible yet operationally weak rules. Sigma-Hunter adapts validated Sigma rules into training examples, keeps each source rule within one data split, and evaluates generated rules across multiple quality dimensions.

  • Threat reports and forensic observations often omit the log sources, fields, filters, and false-positive controls needed for operational rules, making translation time-consuming.Source: §I · p. 1
  • Sigma’s platform-agnostic YAML expresses detection intent while retaining log-source requirements, conditions, severity, false-positive context, and optional MITRE ATT&CK mappings,.Source: §I · p. 1
  • Sigma-Hunter is a domain-adapted LLM trained on question-answer and analyst-reasoning examples programmatically derived from validated open-source Sigma rules.Source: §I · p. 1
  • Each original Sigma rule is assigned to one train, validation, or test split before example expansion, preventing derived tasks from crossing splits.Source: §I · p. 1
  • Untuned Mistral scored 4.61, while syntax validity tracked semantic rule quality only loosely.Source: §I · p. 2
  • Syntax validity tracks semantic rule quality only loosely, motivating evaluation beyond formatting.The evaluation covers detection logic, completeness, selectivity, field consistency, and log-source alignment.Source: §I · pp. 1–2

II. BACKGROUND AND MOTIVATION

Prior work applies LLMs to cyber operations and structured threat artifacts, including security-rule generation. Sigma-Hunter emphasizes source-rule-level leakage control, matched tuned/untuned comparisons, Sigma-specific evaluation, and compact local deployment.

A. Related Work

  • Prior LLM work includes autonomous exploitation of one-day vulnerabilities [3], agent teams for zero-day scenarios [4], and reported offensive and dual-use capabilities in a frontier model.Source: A. Related Work · p. 2
  • Related efforts structure threat intelligence or generate detections: IntelEX, LLMCloudHunter, RulePilot, SigmaGen, CTI-REALM, and GenTI.These efforts motivate evaluation of schema adherence, rule correctness, field validity, and operational usefulness.Source: A. Related Work · p. 2
  • Sigma-Hunter extends this work with source-rule-level leakage control, matched tuned/untuned comparisons, quantitative Sigma-specific evaluation, and compact local deployment.The paper does not propose a new model architecture; local deployment suits air-gapped settings and governance constraints.Source: A. Related Work · p. 2

B. Detection Engineering and Sigma Rules

Threat information does not itself specify a deployable detection: analysts must select available telemetry, relevant fields and patterns, and a scope that avoids excessive false positives. Sigma offers a portable representation, but portability does not guarantee operational quality.

B. Detection Engineering and Sigma Rules

C. Language Models for Detection Engineering

LLMs can help draft and explain detection logic, but general-purpose outputs may violate detection-engineering constraints. Sigma-Hunter uses a staged pipeline to turn validated rules into locally deployable models, with analysts reviewing candidate rules before deployment.

C. Language Models for Detection Engineering

  • LLMs can draft rules, explain logic, map ATT&CK techniques, suggest false-positive considerations, and propose hunting hypotheses.Source: C. Language Models for Detection… · p. 2
  • General-purpose models may produce invalid YAML, missing conditions, wrong log sources, hallucinated fields, or overly broad logic; validated-artifact fine-tuning teaches recurring Sigma conventions.Source: C. Language Models for Detection… · p. 2

III. SIGMA-HUNTER SYSTEM OVERVIEW

  • The pipeline collects Sigma rules and validates their YAML and required fields, removing malformed or unsuitable rules before training.Source: §III · pp. 2–3, Figure 1 · p. 3
  • Validated rules become QAR instruction examples for generation, explanation, ATT&CK mapping, false-positive analysis, hunting guidance, log-source reasoning, and refinement.Source: §III · p. 3
  • The system uses LoRA fine-tuning and exports deployment-friendly formats for local inference, keeping prompts, threat reports, logs, and detection artifacts in operator-controlled environments.Source: §III · p. 3
  • Sigma-Hunter returns candidate rules for analyst review; analysts check local telemetry and field mappings, test historical data, tune filters, and approve rules before deployment.Source: §III · p. 3

A. Rule Collection and Validation

The authors curated and validated open-source Sigma rules, then expanded them into instruction examples for model training and evaluation. The corpus is weighted toward generic detection rules.

A. Rule Collection and Validation

  • 3,635 source Sigma rules were expanded into 7,663 QAR examples, with 6,130 training, 766 validation, and 767 test examples.The rule-generation evaluation draws on 188 examples from the test partition.Source: A. Rule Collection and Validation · p. 3
  • Generic detection rules make up 84.5% of source rules, potentially limiting relative gains on emerging-threat and threat-hunting tasks.Source: A. Rule Collection and Validation · p. 3
  • Rules were retained only if they parsed as YAML and included a valid title, logsource, detection, and detection condition.Rules failing structural checks, length limits, or control-character checks were removed.Source: A. Rule Collection and Validation · p. 3

B. QAR Instruction Example Generation

Validated Sigma rules were converted programmatically into instruction examples, including guided rule generation and auxiliary analyst tasks. The templates derive descriptions from source-rule metadata.

B. QAR Instruction Example Generation

  • Rule-generation prompts pair threat summaries, ATT&CK tags, detection hints, and intended log sources with a complete Sigma rule in YAML.This evaluates guided generation from partial analyst-supplied context, not unconstrained generation from threat reports.Source: B. QAR Instruction Example Generation · pp. 3–4, Table II · p. 4
  • Auxiliary examples cover rule explanation, ATT&CK mapping, false-positive analysis, and threat-hunting guidance.Descriptions come from source-rule metadata rather than an external generative model.Source: B. QAR Instruction Example Generation · p. 4
  • Generated examples were checked for length, empty fields, and problematic characters to reduce malformed inputs and invalid YAML output patterns during training.Source: B. QAR Instruction Example Generation · p. 4

C. Rule-Level Data Splitting

The split assigns each original Sigma rule to one partition before expanding rules into QAR examples, preventing examples derived from the same rule from crossing splits.

C. Rule-Level Data Splitting

V. MODEL FINE-TUNING AND DEPLOYMENT

Sigma-Hunter fine-tunes Mistral and Phi4 variants for detection-engineering tasks and packages the adapted models for local inference. The Mistral training setup uses LoRA and a learning-rate range test.

V. MODEL FINE-TUNING AND DEPLOYMENT

  • Sigma-Hunter-Mistral and Sigma-Hunter-Phi4 were trained to test domain-specific fine-tuning across model families.Source: §V · p. 4
  • The fine-tuned Sigma-Hunter-Mistral uses 41,943,040 trainable parameters, or 0.575% of its approximately 7.29B total parameters.LoRA adapters target attention and MLP projections across all 32 transformer blocks.Source: §V · p. 4, Table III · p. 4
  • Training used three epochs, 4-bit quantization, cosine learning-rate decay, and checkpoint selection by lowest validation loss.After training, adapters were merged with the base model and exported as a GGUF/Ollama package for local deployment.Source: §V · p. 4
  • The Mistral learning-rate range test found steepest loss descent near 3.55×10^-5 and a more conservative point near 1.18×10^-5.The authors favored a stable value below steepest descent and avoided the upper-range instability.Source: Figure 2 · p. 4, §V · p. 5

VI. EVALUATIONS

Sigma-Hunter-Mistral leads the held-out rule-generation benchmark, while the evaluation shows that syntactic validity and plausible fields do not ensure strong detection logic. The benchmark combines structural checks, approximate field consistency, and LLM-assisted semantic judgments.

VI. EVALUATIONS

  • The quantitative benchmark used 188 held-out rule-generation examples, with identical prompts across domain-adapted models and untuned general-purpose baselines.Semantic scores were averaged over successfully judged outputs; non-generation test examples were used only for qualitative analysis.Source: §VI · p. 5
  • Syntax validation scores five structural elements, approximate field consistency compares fields with curated log-source dictionaries, and semantic judging scores detection behavior across five dimensions.Field consistency is approximate because field availability varies by environment; semantic selectivity is judged rather than measured on labeled telemetry.Source: §VI · p. 5
  • The semantic scores are comparative assessments, not independent expert validation; human detection-engineer validation remains future work.The authors used gpt-oss:20b as their strongest locally deployable semantic judge.Source: §VI · p. 5
  • 8.17 overall for Sigma-Hunter-Mistral versus 4.61 for untuned Mistral; Sigma-Hunter-Phi4 scored 8.03 versus 6.82 for untuned Phi4.Matched-model gains also improved detection logic, completeness, and selectivity, not only output formatting.Source: §VI · p. 5, Table IV · p. 6
  • Sigma-Hunter-Mistral preserved both curl.exe and file:/// constraints, while gpt-oss:20b retained only the protocol-handler condition despite both rules scoring 10 for syntax and field consistency.The omission reduced gpt-oss:20b’s selectivity and overall scores to 4 and 5, respectively, versus 10 on both dimensions for Sigma-Hunter-Mistral.Source: §VI · pp. 5–6, Table V · p. 6

VII. CONCLUSION

Sigma-Hunter packages domain-adapted models for local analyst assistance in threat hunting and Sigma rule generation. In evaluation, Sigma-Hunter-Mistral achieved the highest mean overall score among evaluated systems, while Sigma-Hunter-Phi4 improved over its untuned baseline.

VII. CONCLUSION

  • Sigma-Hunter-Mistral achieved the highest mean overall score among evaluated systems, and Sigma-Hunter-Phi4 improved over its untuned baseline.The paper concludes that domain adaptation improves reference-aligned semantic quality beyond syntactic correctness.Source: §VII · p. 6
  • The locally generated drafts are analyst-directed: they are validated against local telemetry, tuned for false positives, and approved before deployment.Future work will add human analyst validation and extend Sigma-Hunter to backend-specific detection languages and operational schemas.Source: §VII · p. 6
Loading 2610.09007v1…