Source-linked AI summary
Evidence-Grounded Retrieval for Investigation Hunt Lead Generation from CTI Reports
Akash Prakash, Boubakr Nour, Makan Pourzandi, Chadi Assi, Mourad Debbabi
TL;DR
AHLERT targets the difficulty of converting unstructured CTI into actionable hunt leads when manual workflows are slow and existing automation overlooks environment constraints. It combines hybrid evidence retrieval, ontology-grounded generation, and structured lead synthesis, raising mean F1 from 0.44 to 0.85 over flat RAG and achieving 86.95% average effectiveness.
Problem
Manual CTI-to-lead construction is slow and hard to scale, while existing approaches often stop at entities, ignore the defender’s environment, and analyze reports in isolation.
Method
AHLERT uses hybrid semantic, graph, and entity retrieval with ontology-aware grounding and evidence-grounded generation to produce prioritized hunt leads.
Results
0.44 to 0.85 mean F1: across four APT reports, AHLERT approximately doubles mean F1 over a single-route flat-RAG baseline and AHLERT-CHATGPT achieves 86.95% average effectiveness.
Takeaways & Limitations
AHLERT produces environment-consistent, prioritized hunt leads rather than loose indicators, and its augmented generators outperform their off-the-shelf counterparts.
Abstract
from arXiv · showhide
Threat hunting increasingly depends on converting unstructured knowledge (e.g., Cyber Threat Intelligence reports) into actionable hunt leads: concise, investigable hypotheses grounded in observable artifacts and adversary techniques. Producing such leads manually is a tedious and hard-to-scale task. Existing automated approaches stop at the entity layer, ignore the defender's operational environment, and analyze each report in isolation. To address these gaps, we introduce AHLERT, a system that automatically extracts relevant, environment-aware, and hunt leads from threat reports through (i) a hybrid retriever that combines dense vector search with multi-hop traversal over a knowledge graph seeded with MITRE ATT&CK; (ii) an ontology-grounding retrieval-augmented generation method that constrains each lead to the defender's own assets and controls; and (iii) an LLM-agnostic framework that emits structured, directly actionable leads rather than loose indicators of compromise. We evaluate AHLERT on public CTI reports for well-known APTs across multiple proprietary and open-weight models. Hybrid evidence retrieval with ontology grounding raises mean F1 by ~2x (0.44 to 0.85) over a single-route flat-RAG baseline, and AHLERT attains the highest effectiveness score (~86.95%) compared with off-the-shelf LLM models.
I. INTRODUCTION
AHLERT addresses the bottleneck of turning unstructured CTI into faithful, environment-aware, investigable hunt leads. It combines hybrid retrieval, ontology grounding, and controlled generation to produce prioritized hypotheses rather than isolated indicators.
- Motivation: Threat hunters must manually distill lengthy CTI reports into actionable leads mapped to their assets and controls, making the process slow, error-prone, and difficult to scale.Reports mix indicators, ATT&CK techniques, malware details, and narrative, while prior systems often stop at entity extraction.
- Challenges: A useful hunt lead must remain faithful to source evidence, reflect the defender’s observable environment, and connect findings through relational knowledge.Unsupported behaviors can misdirect hunts, while absent sensors or isolated report analysis limit operational value.
- AHLERT: AHLERT combines dense retrieval with cyber knowledge graph reasoning to enrich report-derived evidence through relationships among techniques, tools, actors, campaigns, assets, and indicators.Its hybrid design supplements topical similarity with structural and relational evidence paths.
- AHLERT: Ontology-aware grounding constrains generated leads to the defender’s asset and control universe, reducing environment-irrelevant hypotheses.This addresses leads that reference sensors, controls, or data sources the SOC does not operate.
- AHLERT: Evidence-grounded generation verifies and ranks candidate leads, producing fully phrased, prioritized hypotheses instead of loose indicators of compromise.The resulting lead can combine a behavioral hypothesis with supporting evidence and concrete artifacts for investigation.
- Results: 0.44 to 0.85 mean F1: AHLERT raises hunt-lead quality by approximately 2× over a single-route flat-RAG baseline and reaches 86.95% average effectiveness.The evaluation spans multiple contemporary proprietary and open-weight LLMs.
A. System Overview
AHLERT transforms unstructured threat reports into ranked, environment-consistent hunt leads through hybrid retrieval, ontology-aware grounding, and model-agnostic generation.
- A. System Overview: AHLERT fuses semantic similarity, graph expansion, and entity expansion into a unified evidence bundle grounded in a defender environment ontology.The system uses topical, structural, and relational retrieval paths rather than a single modality.
- A. System Overview: AHLERT constrains generated leads to the defender’s environment and prunes leads that do not remain operationally actionable.
- A. System Overview: The generation framework is decoupled from retrieval behind a uniform interface, supporting unchanged execution across proprietary and open-weight models.
B. Offline Knowledge Construction
AHLERT builds a persistent CTI knowledge substrate that combines a semantic index, a relationship graph, and provenance to support retrieval across historical intelligence and cyber entities.
- B. Offline Knowledge Construction: AHLERT ingests historical CTI and a curated cyber knowledge base into a persistent substrate exposing both semantic and graph views with item provenance.
- B. Offline Knowledge Construction: The CTI graph links threat groups, techniques, software, reports, and their relationships using MITRE ATT&CK and historical incident reports.New reports are incrementally merged so retrieval can connect observations to prior intelligence and reach known actor techniques in a few hops.
- B. Offline Knowledge Construction: The semantic index segments documents into overlapping 600-token passages with an 80-token overlap for meaning-based passage recall.
- B. Offline Knowledge Construction: Each passage is encoded into a 384-dimensional normalized vector and indexed with FAISS HNSW for approximate nearest-neighbor search with provenance metadata.
C. Online CTI Report Analysis
Online analysis normalizes a submitted report, extracts structured evidence units, and enriches them through semantic, graph, and entity retrieval routes.
- C. Online CTI Report Analysis: AHLERT normalizes raw reports through extraction, cleaning, segmentation, and near-duplicate removal before producing clean report segments.
- C. Online CTI Report Analysis: Evidence-unit extraction captures each passage’s entity, behavior, technique, tool, infrastructure, target, and temporal context.
- C. Online CTI Report Analysis: Hybrid retrieval uses semantic similarity to recall related passages, graph expansion to follow cyber relationships, and entity expansion to retrieve connected actors, assets, and indicators.
- C. Online CTI Report Analysis: In the illustrative example, semantic retrieval recalls passages at cosine similarities 0.89 and 0.84, graph traversal reaches APT41 and related techniques, and entity expansion retrieves POISONPLUG.
D. Hunt Lead Generation
AHLERT converts retrieved evidence into ranked hunt leads, verifies them for support and environmental consistency, removes redundancy, and prioritizes validated hypotheses.
- D. Hunt Lead Generation: The final phase produces concise, action-oriented hypotheses that threat hunters can execute directly against enterprise telemetry.
- D. Hunt Lead Generation: The synthesizer combines report fragments, ATT&CK techniques, graph context, provenance, and the analyst’s task objective to draft candidate leads.
- D. Hunt Lead Generation: Verification discards unsupported leads, prunes operationally infeasible leads, and merges near-duplicate candidates before ranking.
- D. Hunt Lead Generation: Surviving leads receive calibrated confidence scores reflecting evidence strength, environmental consistency, and activity severity, with severity breaking ties.
III. EXPERIMENTAL SETUP
The experimental setup combines a multi-model evaluation, public APT41 threat reports, expert-curated hunt-lead ground truth, and cyber and system ontologies.
- Evaluation configurations: The prototype uses multiple proprietary and open-weight generators, including ChatGPT, Qwen, and Foundation-Sec, with corresponding AHLERT and off-the-shelf configurations.Six configurations span three AHLERT-augmented generators and their off-the-shelf counterparts.
- Threat reports: The evaluation uses four public APT41 reports from the APTNotes repository, sourced from vendors including Mandiant, Trend Micro, and CrowdStrike.
- Ground truth: An expert-curated ground-truth set covers investigable leads across initial access, execution, persistence, and exfiltration.Each ground-truth lead follows the same imperative-sentence schema as AHLERT outputs.
- Ontologies: The cybersecurity ontology contains 12,838 triples and 4,280 entities spanning APT groups, malware families, and CVE vulnerabilities.It includes 103 APT groups, 296 malware families, 2,896 CVE vulnerabilities, and 115 relationship types.
- Ontologies: The system ontology represents 79 nodes across virtual machines, servers, firewalls, applications, services, and workstations.
IV. EVALUATION
AHLERT is evaluated for lead relevance, contextual correctness, actionability, retrieval and grounding effects, and validation across generators. The results show higher-quality, more environment-grounded leads than baseline configurations, while the authors identify limitations in faithfulness benchmarking and evaluation design.
- Effectiveness: AHLERT-CHATGPT achieves the highest effectiveness score across four reports, averaging 86.95%.The score combines lead relevance, IoC accuracy, and actionability.
- Validation: Human and GPT-4o validation agree that AHLERT configurations outperform their off-the-shelf counterparts, although AHLERT-CHATGPT also supplied the ground-truth reference.For open-weight pairs, gains are consistently about three percentage points under either validator.
- Contextual correctness: AHLERT-CHATGPT reaches a 97.2% grounding rate, exceeding CHATGPT-OTS at 95.8%.Grounding requires at least one lead entity to match an entity declared in the defender ontology.
- Actionability: AHLERT-CHATGPT attains an 88% actionability score, compared with 64% for CHATGPT-OTS.The score requires an imperative verb, an ATT&CK technique or CVE, and concrete scoping parameters.
- Retrieval ablation: Hybrid retrieval raises mean F1 from 0.44 for flat-RAG to 0.85, approximately doubling performance.The comparison disables graph and entity expansion in the flat-RAG baseline.
- Ontology ablation: Ontology grounding lifts mean F1 from 0.64 to 0.85 on hybrid graph retrieval, a gain of 0.21.The improvement is reported while holding the retriever fixed.
- Limitations: A dedicated faithfulness benchmark remains future work, despite verification that discards leads unsupported by retrieved evidence.The paper also reports that 70%–97% of leads reference assets in the defender ontology.
V. RELATED WORK
Prior CTI systems commonly stop at entity or technique extraction, whereas AHLERT uses structured extraction to build ranked, environment-consistent hunt hypotheses. Its hybrid retrieval combines semantic similarity, graph traversal, and entity expansion into an evidence bundle grounded by a system ontology.
- Most prior CTI extraction efforts stop at the entity or technique-mapping layer, leaving SOC analysts to hand-craft hunt queries.
- AHLERT treats structured extraction as an intermediate means for generating ranked, environment-consistent hypotheses rather than static extraction bundles.
- Existing CTI-RAG systems commonly stop at technique annotation, free-form question answering, or graph construction.
- AHLERT fuses semantic similarity, graph traversal, and entity expansion into one evidence bundle grounded through a system ontology.
VI. CONCLUSION
AHLERT automatically extracts actionable, evidence-grounded hunt leads from unstructured CTI reports using hybrid retrieval, generation, and environment filtering. Across four APT reports, it substantially outperformed a flat-RAG baseline and off-the-shelf generators.
- AHLERT combines flat retrieval, graph traversal, and controlled generation while filtering leads against the defender’s infrastructure.
- 0.44 to 0.85 mean F1: the full hybrid pipeline achieved approximately 2× the single-route flat-RAG baseline across four APT reports.
- 86.95% average effectiveness: the AHLERT-augmented GPT achieved the highest effectiveness score.
- All three AHLERT-augmented generators exceeded their off-the-shelf counterparts.
MODEL-AWARE PROMPTING
AHLERT prompts the model to transform a retrieved evidence bundle into ranked hunt leads grounded in both CTI evidence and the defender’s declared environment.
- The prompt asks a senior threat-hunting analyst to produce concrete, investigable hunt leads for the SOC’s own telemetry.
- Each lead must be ranked and grounded in matched CTI passages, ATT&CK techniques, graph context, and system-ontology entities.