PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs for Zero-Shot Threat Detection via Structured Risk Indicators

Abdullah Alghamdi, Siamak Layeghy, Marius Portmann

arXiv:2608.16508v1cs.CRcs.LGcs.NI

TL;DR

Detecting insider threats and APTs is difficult because malicious actions can appear benign individually and unfold across heterogeneous logs. This paper uses a two-stage LLM framework with personalised retrieval and structured risk indicators, outperforming GABM by 11.40 and 31.50 F1-score percentage points on CERT r5.2 and PicoDomain, respectively.

  • Problem

    Threat detection requires contextual reasoning across heterogeneous logs, historical behaviour, and attack stages because individual malicious actions often appear benign in isolation.

  • Method

    A two-stage LLM framework organises logs into user timelines, generates structured risk indicators, and classifies their temporal evolution with optional personalised retrieval.

  • Results

    11.40 and 31.50 percentage-point F1-score improvements over GABM are achieved on CERT r5.2 and PicoDomain, respectively, with every evaluated configuration outperforming the baseline.

  • Takeaways & Limitations

    Retrieval and model capability appear to act as substitutes, while the most effective model assignment across stages depends on the dataset.

  • Takeaways & Limitations

    Absolute metrics cannot be compared directly with GABM because its classification unit, labelling, and benign-sampling procedure differ from this framework’s.

Abstract

from arXiv · show

We propose a two-stage large language model (LLM) framework for zero-shot detection of insider threats and advanced persistent threats (APTs) from heterogeneous security logs. The framework models user activity as chronological timelines and incorporates retrieval-augmented generation (RAG) to provide personalised behavioural context from each user's historical activity. Rather than performing end-to-end classification directly from raw logs, it first generates structured, interpretable sets of threat-specific risk indicators, which are then classified jointly across temporal sequences to capture attack patterns spanning multiple windows.The framework is evaluated on two benchmark datasets, CERT r5.2 for insider threat detection and PicoDomain for APT detection, using four combinations of two open-weight LLMs under both retrieval and non-retrieval settings. All configurations outperform the previous state-of-the-art LLM-based framework (GABM), with the best configuration improving the F1-score by 11.40 percentage points on CERT r5.2 and 31.50 percentage points on PicoDomain. Results further show that retrieval mainly benefits weaker LLMs by generating more discriminative risk indicators, whereas stronger models achieve comparable performance without retrieved context. The most effective assignment of LLMs to the two stages depends on the dataset. These findings show that the quality of the generated risk indicators is the main driver of zero-shot cyber threat detection performance.

1. Introduction

The paper proposes a two-stage zero-shot LLM framework that generates structured risk indicators before temporal classification, using timelines and personalised retrieval to detect insider threats and APTs. Across CERT r5.2 and PicoDomain, all evaluated configurations outperform GABM, while retrieval and model assignment affect performance by dataset and model capability.

  • Framework motivation and contribution: Chronological user timelines unify multiple log sources, while retrieval-augmented generation supplies personalised context from semantically similar historical activity.Historical behaviour helps distinguish normal variation from malicious activity while preserving zero-shot detection.
  • Evaluation and findings: The evaluation covers CERT r5.2 for insider threats and PicoDomain for APTs across four combinations of two open-weight LLMs with and without retrieval augmentation.CERT r5.2 uses host activity logs, whereas PicoDomain uses Zeek network logs.
  • Evaluation and findings: Every evaluated configuration outperforms GABM, while retrieval primarily benefits weaker LLMs by improving risk-indicator quality and optimal stage-model assignment varies by dataset.The findings identify generated risk-indicator quality as central to detection performance.
  • Framework motivation and contribution: The framework replaces direct classification of heterogeneous security logs with structured risk-indicator generation followed by temporal classification for unified insider-threat and APT detection.This abstraction is intended to make behavioural evidence interpretable and capture attack patterns across temporal evolution.

2. Related Work

Prior LLM-based cyber threat detection enables zero-shot reasoning over heterogeneous logs but generally performs end-to-end classification from raw or lightly processed data. The proposed work instead uses personalised historical behaviour as retrieval context, positioning retrieval as behavioural grounding rather than external knowledge augmentation.

  • LLM-based threat detection: Recent LLM-based approaches shift cybersecurity log analysis toward zero-shot reasoning over heterogeneous security logs without task-specific training.Existing work explores prompt engineering, parameter-efficient fine-tuning, and multi-view behavioural modelling.
  • Limitations of prior work: Most existing approaches classify threats end-to-end from raw or lightly processed logs, requiring one inference step to interpret events, reason about behaviour, and decide.The paper reports that this design produces unstable predictions and low precision in preliminary experiments, consistent with Xu et al. (2024).
  • Closest frameworks: GABM and Audit-LLM are the closest related multi-agent detection frameworks; GABM is the benchmark state of the art but trades low precision for high recall.Audit-LLM also coordinates threat analysis through multiple agents and reports user-level evaluation metrics.
  • Retrieval-augmented generation: Unlike cybersecurity RAG systems that retrieve external threat intelligence, this work retrieves each user’s historical activity to provide personalised behavioural context.The retrieval compares each activity window with the same user’s history, shifting retrieval from knowledge augmentation to behavioural grounding.
  • Personalised behavioural context: The proposed framework generates risk indicators relative to an individual’s established activity patterns rather than relying on generic notions of suspicious behaviour.This personalised baseline is presented as a distinction from prior approaches that do not explicitly model user behaviour relative to historical activity.

3. Methodology

The framework combines temporal windowing, behaviour-aware retrieval, and dual-LLM inference for feature generation and classification over ordered windows. It also samples windows deterministically for LLM inference and compares RAG with No-RAG configurations.

  • Framework components: The framework comprises data preparation and temporal windowing, behaviour-aware retrieval for contextual augmentation, and dual-LLM inference over ordered windows.The two LLM stages perform feature generation and classification.
  • Inference sampling: A deterministic temporal-context sampling step selects which windows undergo LLM inference because the windowed datasets are large.The sampling step is specified in Section 3.2.5.
  • Evaluation configurations: The framework is evaluated under RAG and No-RAG configurations to isolate retrieval’s contribution across both models.Figure 1 provides an overview of the pipeline.

3.1. Data Preparation and Temporal Windowing

The framework consolidates each user’s heterogeneous security records into a chronological timeline, then applies stride-1 sliding windows with an any-positive malicious label. Each window is encoded compactly for embeddings and richly for LLM-based analysis.

  • Timeline construction: User activity is grouped and chronologically ordered so cross-source patterns, rather than isolated records, provide each inference window with complete behavioural context.CERT r5.2 merges logon, device, file, email, and HTTP sources; PicoDomain links log entries to users through dataset-provided host-to-user mappings.
  • Temporal windowing: Sliding windows use stride 1, with w=10 for CERT r5.2 and w=5 for PicoDomain.A window is malicious if any constituent log entry is malicious; under stride 1, one malicious entry can label up to w consecutive windows.
  • Window representation: Each window is encoded as a compact protocol-and-destination token sequence for the embedding model and as richer full Zeek records for LLM1.Destinations are categorized as internal, known C2, or external in the compact representation.

3.2. Behaviour-Aware Retrieval for Contextual Augmentation

The framework augments each window with personalised historical context retrieved from the same user’s timeline before LLM1 generates risk indicators. Deterministic sampling preserves malicious bursts and benign transitions while ensuring fair comparisons, though reported metrics reflect the sampled distribution.

  • Retrieval mechanism: RAG retrieves similar past windows from the same user’s history, enabling LLM1 to compare current behaviour against a personalised historical baseline.Retrieved context is attached to each window before risk-indicator generation.
  • Retrieval mechanism: k=3 similar windows are retrieved from a preceding 500-window same-user horizon using cosine similarity over 384-dimensional all-MiniLM-L6-v2 embeddings.Embeddings are computed once per window and reused across downstream steps.
  • Retrieval mechanism: The No-RAG condition removes retrieved windows and deterministic similarity features, requiring LLM1 to generate risk indicators from the current window alone.All other pipeline components remain identical, isolating retrieval’s contribution.
  • Inference sampling: 2,741 sampled windows are retained for PicoDomain with ±10 benign-window context around all malicious bursts, while CERT r5.2 retains 9,571 sampled windows using the largest burst per user and ±30 context.Sampling preserves temporal transitions between normal and anomalous behaviour.
  • Inference sampling: Reported precision and F1 reflect the burst-focused sampled distribution rather than the full-corpus base rate, although identical subsets preserve relative comparisons across conditions.The deterministic procedure uses window labels and sequential positions only.

3.3. Feature Generation (LLM1)

LLM1 replaces hand-engineered extractors by converting each log window into threat-specific, normalised risk indicators for downstream sequence classification. With retrieval enabled, it compares the current window with up to three similar past windows from the same user using similarity features.

  • Feature generation: The two-stage process uses LLM1 to generate ordered feature vectors and LLM2 to classify sequences of those vectors.This replaces conventional hand-engineered feature extraction with a threat-aware LLM component.
  • Feature generation: LLM1 transforms each sampled log window into a structured set of normalised risk indicators tailored to the dataset’s threat type.The current window’s log content is provided without labels.
  • Retrieval-augmented generation: In retrieval-augmented inference, LLM1 receives up to three past windows from the same user plus deterministic similarity features measuring current-window anomaly relative to recent history.The prompt directs LLM1 to compare the current window with retrieved past windows.
  • Dataset-specific feature sets: Feature schemas satisfy coverage and extractability: every documented threat-scenario dimension must be observable, and each feature must be derivable from available log types.The schemas are dataset-specific and use scenario documentation only to define the feature schema.
  • Dataset-specific feature sets: PicoDomain features target seven APT categories, including connection-pattern anomalies, authentication abuse, lateral movement, protocol irregularities, exfiltration, attack progression, and aggregate risk.The features are extractable from CONN, DNS, DCE_RPC, SSL, and Kerberos logs.

3.4. Sequence Classification (LLM2)

LLM2 classifies each chronological user-activity window as benign or malicious using only the risk-indicator vectors generated by LLM1. It reasons over temporal sequences, using overlapping chunks and surrounding context to identify burst boundaries and coordinated attack patterns.

  • Sequence input and labeling: LLM2 assigns a binary benign-or-malicious label to every window from LLM1’s chronological risk-indicator vectors, without receiving raw logs or downstream similarity features.The labels are defined as 0=benign and 1=malicious.
  • Chunked processing: The ordered feature sequence is divided into chunks of 20 windows with an overlap of 5 windows to accommodate context-window constraints.For windows appearing in multiple chunks, the later chunk’s prediction is retained because it includes subsequent context.
  • Burst-aware reasoning: LLM2 reasons over full temporal sequences rather than isolated windows, identifying elevated-risk regions and coordinated patterns such as data exfiltration, privilege abuse, and suspicious access.Surrounding context is used to classify ambiguous windows, and pattern descriptions are instantiated per dataset.

4. Experimental Setup

The framework is evaluated on PicoDomain and CERT r5.2 using precision, recall, and F1-score aggregated across users, with GABM as the primary comparable LLM-based baseline. Experiments also test direct single-pass LLM classification, whose limited performance motivates the proposed structured, multi-stage design.

  • Datasets: PicoDomain is a three-day Zeek network-log dataset simulating an enterprise under an advanced persistent threat campaign.It contains multiple Zeek log types and is evaluated after preprocessing user-associated activity.
  • Datasets: CERT r5.2 is an 18-month synthetic organisation dataset containing 2,000 employees and 99 employees with labelled malicious activity across four insider-threat scenarios.This work uses logon, device, file, email, and HTTP logs from the seven available log types.
  • Baseline: The proposed framework is compared against GABM, a recent LLM-based multi-agent insider-threat method evaluated on the same two benchmark datasets.GABM uses specialised agents for each log type and a supervisor agent for final classification with LLaMA-3.1-8B.
  • Evaluation: Performance is measured with precision, recall, and F1-score aggregated across all users in each dataset, with F1 as the primary comparison basis.Aggregation covers 5 PicoDomain users and 99 CERT r5.2 users.
  • Ablation: 30.98% and 23.10% are the best matched-zero-shot direct-classification F1 scores for DeepSeek-R1-Distill-Qwen 32B and Llama 3.1 8B, respectively, on CERT r5.2.No combination of shot count and batch size raises direct classification beyond a low ceiling, and all configurations remain below the proposed framework.
  • Ablation: A single LLM cannot perform the task end-to-end across both models and datasets, motivating separate feature generation and sequence-classification stages.The direct prompt must parse heterogeneous logs, reason about threat behaviour, and make the final decision simultaneously.

5. Results and Discussion

The framework outperforms GABM across both datasets, with performance driven primarily by LLM1’s structured risk indicators and user-specific behavioural grounding. Results also show dataset-dependent model-stage effects, heterogeneous user performance, and important evaluation limitations.

  • Model and retrieval effects: LLM1 selection affects performance more than LLM2 selection, while retrieval mainly helps the weaker Llama 3.1 8B feature generator.The strongest model assignment is dataset-dependent: classification benefits most from the larger model on CERT r5.2, whereas feature generation does so on PicoDomain.
  • Model and retrieval effects: Retrieval improves selectivity on CERT r5.2: under DeepSeek/DeepSeek, false positives fall from 4,490 to 3,447 and specificity rises from 13.0% to 33.2%.Retrieved per-user behavioural context calibrates risk scores against historical baselines, although the No-RAG configuration achieves higher F1 through a recall-biased operating point.
  • Overall performance: 64.14% F1 on CERT r5.2 and 50.87% on PicoDomain are the framework’s best results, improving over GABM by 11.40 and 31.50 percentage points, respectively.The strongest CERT configuration uses Llama 3.1 8B as LLM1, DeepSeek as LLM2, and RAG; PicoDomain uses DeepSeek as LLM1, Llama 3.1 8B as LLM2, and No-RAG.
  • Feature quality and dataset differences: PicoDomain risk indicators show weak class separation, with mean absolute class-average difference 0.008 and MCC = 0.004, versus MCC = 0.146 on CERT r5.2.On PicoDomain, feature variance and activation counts are nearly identical across classes, and some benign windows score higher than malicious ones.
  • Representation and grouping ablations: Structured event objects outperform verbose full-text and compact sequence-and-destination representations, providing the strongest LLM1 signal on CERT r5.2.User-level timelines also outperform connection-pair grouping on PicoDomain, reaching F1 = 45.74% versus 29.87% and preserving broader behavioural context.

6. Conclusion

The retrieval-augmented dual-LLM framework converts per-user chronological log windows into interpretable risk indicators and classifies their ordered sequences for zero-shot intrusion detection. It outperforms GABM across CERT r5.2 and PicoDomain, while retrieval benefits weaker models and optimal stage assignments vary by dataset.

  • Framework: The framework groups heterogeneous logs into per-user chronological timelines, generates threat-specific structured risk-indicator vectors with LLM1, and classifies ordered vector sequences with LLM2.This two-stage design targets multi-window attack patterns rather than direct classification from raw logs.
  • Results: 64.14% and 50.87% are the best F1-scores on CERT r5.2 and PicoDomain, improving over the GABM baseline by 11.40 and 31.50 percentage points, respectively.All eight evaluated configurations exceed GABM on both datasets.
  • Cross-model analysis: Retrieval and model capability act as substitutes: retrieval improves weaker models, whereas the more capable model attains comparable feature quality without retrieved context.The conclusion attributes retrieval’s main benefit to generating more discriminative indicators for weaker models.
  • Cross-model analysis: The best model assignment differs by dataset, placing the more capable model at feature generation on weak-signal data and at classification on strong-signal data.This assignment pattern links stage specialization to the strength of the dataset’s threat signal.
  • Feature analysis: CERT r5.2 shows clearer behavioural traces than PicoDomain, with mean absolute class differences of 0.057 across 17 features versus 0.008 for stealthy APT beaconing.Despite PicoDomain’s limited discriminative signal, the structured representation provides a consistent basis for LLM2’s window-level classification.
  • Future work: Future work will evaluate additional datasets, investigate cross-user retrieval for shared C2 beaconing patterns, and explore fine-tuning LLMs for feature generation.These directions extend the framework’s evaluation, retrieval scope, and feature-generation capability.
Loading 2608.16508v1…