PaperlayerView exact revision on arXiv

Source-linked AI summary

A SoK for SoCs: Reading the TI Leaves on AI for Cyber Threat Intelligence Generation and Sharing

Saastha Vasan, Hadjer Benkraouda, Jizhou Chen, Leyan Pan, Doguhan Yeke, Shinan Liu, Noah Spahn, Stefano Ortolani, David Evans, Christopher Kruegel, Giovanni Vigna

arXiv:2609.01174v1cs.CR

TL;DR

The paper addresses the understudied process of transforming raw attack evidence into shareable cyber threat intelligence. It surveys the literature and practitioners, structures generation and sharing into four steps, and pilots LLMs across them. LLMs can assist each step, but incomplete extraction, weak evidence grounding, and unresolved usefulness judgments mean expert supervision remains necessary.

  • Problem

    The process of converting local attack evidence into shareable CTI is understudied, although errors in extraction, redaction, taxonomy mapping, and format translation are costly.

  • Method

    The authors survey 123 papers and practitioners, define four CTI Generation and Sharing steps, and evaluate three LLMs through four aligned pilot studies.

  • Results

    LLMs assist across all four steps, but recover only a quarter to a third of labeled indicators, struggle to ground claims, and cannot judge report usefulness.

  • Takeaways & Limitations

    Automating shareable CTI requires decomposed workflows, evidence-linked validation, explicit redaction policies, and benchmarks spanning raw evidence through shared intelligence.

  • Takeaways & Limitations

    Existing datasets lack the inputs and labels required for these pilot studies, so the authors build evidence and labels from 49 malware samples and 5 replayed campaigns.

Abstract

from arXiv · show

Cyber Threat Intelligence (CTI) is essential for defending mission-critical infrastructure, yet the process of transforming raw attack evidence into shareable CTI remains fragmented and understudied. We conduct a literature survey of academic papers, organizing the CTI lifecycle into three stages: Threat Data Collection, CTI Generation and Sharing, and CTI Consumption. The first and third stages are well represented in the literature, whereas only a small number of papers address CTI Generation and Sharing. To learn how this stage is practiced, we survey practitioners across multiple organizations who routinely generate and share CTI. They describe a largely manual process with four recurring challenges: preventing the exposure of sensitive information, extracting indicators from noisy attack data, correlating observed behavior with standardized tactics, techniques, and procedures (TTPs), and translating CTI into the formats that sharing platforms require. Using the insights from the practitioner survey, we divide the CTI Generation and Sharing stage into four steps: Intelligence Extraction, Normalization and Enrichment, Codification, and Distribution. We then conduct pilot studies that probe the feasibility of current Large Language Models (LLMs) for each step. The pilot studies show that LLMs can assist an analyst in each of the four steps. However, the models recover only a fraction of the indicators the evidence contains, struggle to ground every claim in the supplied evidence, and do not judge what keeps shared intelligence useful to its recipients. Each step therefore requires expert supervision. Based on these observations, we derive three research directions for automating the production of shareable intelligence.

1 Introduction

CTI enables collaborative defense, but transforming local attack evidence into shareable intelligence remains understudied and largely manual. The paper surveys this gap, studies practitioner workflows, and pilots LLM support across four generation and sharing steps.

  • 123 papers reveal a major research imbalance: only 16 address CTI Generation and Sharing, compared with 44 on collection and 71 on consumption.
  • Practitioners report four recurring challenges: protecting sensitive information, extracting indicators from noise, mapping behavior to TTPs, and translating intelligence into sharing formats.
  • The authors structure CTI Generation and Sharing into Intelligence Extraction, Normalization and Enrichment, Codification, and Distribution.
  • Four pilot studies test three LLMs on indicator extraction, ATT&CK mapping, PII/OII redaction, and STIX 2.1 conversion.
  • LLMs assist across the steps but recover only some indicators, struggle to ground claims in evidence, and require analyst review.

2 Literature Review and CTI Lifecycle

The paper organizes the CTI lifecycle into collection, generation and sharing, and consumption, then shows that research concentrates on the endpoints. CTI Generation and Sharing is routinely performed in practice but remains manual, difficult to reproduce, and dependent on expert judgment.

  • The CTI lifecycle comprises Threat Data Collection, CTI Generation and Sharing, and CTI Consumption.
  • Threat Data Collection converts raw telemetry into structured local representations, while CTI Consumption validates, prioritizes, and applies intelligence shared by others.
  • The middle stage transforms localized evidence into shareable CTI through extraction, normalization and enrichment, codification, and distribution.
  • 44 papers address Threat Data Collection, 16 address CTI Generation and Sharing, and 71 address CTI Consumption.
  • Most generation-and-sharing papers evaluate already-shared intelligence or establish standards rather than explain how shareable intelligence is produced.
  • The scarcity of research does not reflect absence in practice; practitioners routinely perform this work, which remains manual, hard to reproduce, and reliant on expert judgment.

3 Practitioner Survey

A survey of 18 practitioners across five organizations finds that CTI generation and sharing remains lightly automated. Respondents consistently identify privacy, noisy data, taxonomy mapping, and format translation as operational challenges and AI targets.

  • 18 participants from 5 organizations described workflows, challenges, and views on LLM support for CTI generation and sharing.
  • Scripted parsing and manual analysis dominate practice, while only 5 of 18 use machine learning and 3 of 18 use LLMs.
  • 17 of 18 participants cite sensitive-information exposure, 13 of 18 noisy logs, 10 of 18 behavior-to-taxonomy mapping, and 7 of 18 format translation as recurring challenges.
  • These four challenges define Intelligence Extraction, Normalization and Enrichment, Codification, and Distribution.
  • Practitioners select the same tasks as leading AI opportunities but remain concerned about reliability and data privacy.

4 Threat Data Collection

Threat Data Collection reconstructs attacks from organizational telemetry and sandbox observations, producing structured evidence for later CTI generation. Despite mature collection systems, their outputs remain environment-specific and mix relevant activity with local artifacts.

  • Organizational logs and sandbox analysis are the two main evidence sources for CTI generation.Sandbox evidence is the most common starting point among surveyed practitioners, used by 17 of 18.
  • Telemetry is condensed into attack timelines, provenance graphs, behavior indicators, and malware sandbox reports.
  • Sandbox execution logs record process, file, registry, and network activity from controlled malware execution.
  • Collection outputs remain tied to their producing environments, containing local hostnames, internal addresses, or analysis-environment artifacts.

5 CTI Generation and Sharing

CTI Generation and Sharing transforms local attack representations into intelligence that can move between organizations through four steps: extraction, normalization and enrichment, codification, and distribution. The workflow remains substantially manual because analysts must interpret evidence, protect sensitive information, apply shared taxonomies, and produce channel-specific formats.

  • The stage transforms locally bound attack timelines, graphs, behavior indicators, forensic paths, and sandbox reports into shareable intelligence.
  • The four steps are Intelligence Extraction, Normalization and Enrichment, Codification, and Distribution.The first two generate CTI from local attack data, while the latter two prepare it for sharing.
  • Intelligence Extraction separates portable attacker artifacts from benign activity in raw evidence, while Normalization and Enrichment map behavior and weaknesses to standards such as ATT&CK, CWE, and CPE.The survey found extraction from noisy logs and correlation with established taxonomies to be recurring practitioner challenges.
  • Codification selects, filters, transforms, and redacts CTI so it remains actionable for recipients while protecting sensitive organizational information.Redaction must distinguish victim details from attacker information because removing too much reduces utility and removing too little creates exposure and legal risk.
  • Distribution converts codified intelligence into formats required by sharing channels, including structured STIX bundles for restricted exchanges.Although 10 of 18 participants used platforms such as MISP, 7 of 18 identified translation across channel formats as an obstacle.
  • Across all four steps, interpretation and textual restructuring remain manual, motivating one LLM experiment for each step.

6 Exploring AI-Driven Workflows for CTI

The paper evaluates three LLMs as exploratory probes for four CTI-generation steps, finding useful assistance alongside incomplete recovery, unsupported claims, and unresolved privacy–utility judgments.

  • Experiment design: The four experiments test indicator extraction, ATT&CK technique mapping, PII/OII redaction, and conversion of findings into STIX 2.1.The evaluation uses three recent LLMs and focuses on whether current models can support each step, not on ranking providers.
  • Intelligence Extraction: LLMs report 363–461 of 1,472 labeled indicators, achieving recall of 0.25–0.31 despite higher precision than static rules.The models reach precision between 0.51 and 0.78, while rules recover every labeled indicator but over-report substantially.
  • Normalization and Enrichment: LLMs achieve ATT&CK-mapping F1 of 0.71–0.72 versus 0.16 for Sigma, but still assert techniques unsupported by the evidence.The models lose little when sub-technique matching is required, yet they can both miss labeled techniques and assert techniques absent from a report.
  • Codification: Prompt goals reverse redaction errors: privacy-first prompting causes over-redaction, whereas utility-first prompting causes identifier leaks.Under privacy prompting, models miss at most 3 of 92 injected values but may remove non-identifying content; under utility prompting, some injected values remain.
  • Implications: An open-weight model can address cost and confidentiality concerns by running on organization-controlled hardware, but expert policies and supervision remain necessary.The paper identifies redaction policy as necessary for balancing privacy and utility and recommends validating claims rather than only output format.
  • Distribution: Every model produces parseable STIX files preserving supplied findings, but generated links exceed evidential support and require validation.GLM 5.2 asserts 73 links with 38 correct, while GPT-5.5 asserts 67 links; software typing is correct for 19 of 21 software items.

7 Discussion and Future Research Directions

The discussion identifies task decomposition, output validation, better end-to-end datasets, broader enrichment, and explicit redaction policies as directions for improving CTI Generation and Sharing.

  • Task Decomposition and Agentic Workflows: Current models need decomposed, evidence-linked workflows because single-pass processing misses indicators and misidentifies techniques.The proposed workflow segments evidence, separately proposes and verifies claims, and attaches supporting events for analyst review.
  • Task Decomposition and Agentic Workflows: Validators should check whether STIX claims are supported, not merely whether generated files satisfy the required format.Unsupported relationships can pass every format check and should be corrected before distribution.
  • Datasets for CTI Generation and Sharing: Future benchmarks should preserve the full path from raw evidence to shared intelligence so errors can be traced across the pipeline.The authors had to construct both evidence and labels because suitable labeled data was unavailable.
  • Datasets for CTI Generation and Sharing: Enrichment evaluations should extend beyond ATT&CK techniques to CVEs, CAPEC patterns, attribution, and severity context.These additions could make resulting CTI more actionable for recipients.
  • Redaction Policies: Redaction requires explicit policies because models follow stated privacy or utility priorities but cannot decide the value of anonymity versus usefulness.Future policies should define identifiability thresholds, removal costs, replacements, and consequence-based over-redaction measures.

8 Conclusion

The paper finds a research-practice gap in CTI Generation and Sharing, which remains manual despite LLM assistance across extraction, enrichment, redaction, and distribution. It concludes that domain-specific workflows, policies, validators, and end-to-end benchmarks are needed rather than autonomous generation.

  • Conclusion: 123 papers show that Threat Data Collection and CTI Consumption are well studied, while CTI Generation and Sharing remains a research gap.Practitioners identify redaction, extraction, and standardization as the primary bottlenecks in this largely manual stage.
  • Conclusion: LLMs produce fewer false indicators than rule-based baselines and recover most labeled ATT&CK techniques where static rules recover almost none.However, they recover only a quarter to a third of labeled indicators and generate unsupported relationships in otherwise valid STIX files.
  • Conclusion: Future research should decompose tasks, tie claims to evidence, apply explicit redaction policies, validate semantic claims, and benchmark the complete pipeline.These directions target errors that recur across all three evaluated models.
  • Conclusion: The open-weight model is competitive at technique mapping and redaction at lower cost, enabling research on hardware organizations control.The authors plan to share their framework, datasets, and experimental methodology.

Ethics Considerations

The ethics analysis considers stakeholders, privacy, dual-use risks, and reproducibility while emphasizing defensive benefits and safeguards. It frames LLM-assisted CTI generation as analyst-supervised rather than autonomous, with proprietary-model evolution limiting exact reproducibility.

  • Stakeholders: The study involves practitioners, CTI-producing organizations, CTI consumers, researchers, and potential adversaries as distinct stakeholder groups.The analysis considers both defensive beneficiaries and the dual-use nature of CTI automation.
  • Respect for persons and privacy: The practitioner survey involved 18 analysts across 5 organizations and collected only high-level workflow descriptions without sensitive operational data.Responses were reported in aggregate to prevent re-identification.
  • Evidence production and malware handling: The evaluation used 49 malware samples and 5 campaign replays in controlled environments, with no production systems involved.The work developed no new malware, vulnerabilities, or exploits and released hashes rather than samples.
  • Dual-use considerations: CTI automation may be misused by adversaries, so the paper emphasizes analyst-in-the-loop workflows and avoids claims of end-to-end autonomous generation.The authors document systematic model errors and state that the work introduces no new attack techniques.
  • Balancing privacy and utility: Redaction is framed as assistive rather than authoritative because models cannot weigh a detail’s value to recipients against what it reveals about victims.The model follows the priority specified in its prompt, but the input contains no policy for resolving that trade-off.
  • Reproducibility caveat: Exact reproducibility is limited because proprietary LLMs evolve, potentially changing both experimental results and observed errors.The authors name model versions, record configurations, and encourage future reevaluation.

Open Science

The paper reports its literature corpus and practitioner survey as reproducible artifacts, while additional experiment materials are released anonymously for reviewers. Some survey and malware data remain restricted to protect participant privacy and avoid redistributing samples.

  • Artifacts Contained in the Paper: The literature corpus contains 123 papers, with assigned lifecycle stages shown in Figure 2.The venues and exact Google Scholar and DBLP queries are listed in Appendix A, enabling independent reproduction of the search.
  • Artifacts Contained in the Paper: The full practitioner questionnaire and aggregated responses from 18 participants are provided in Table 5.The survey artifact is reported directly in the manuscript.
  • Artifacts Released Anonymously: The anonymous reviewer repository contains prompts, datasets and labels, model outputs, and experiment scripts.The repository link is masked in the preprint and reserved for reviewers of the submitted version.
  • Limitations on Release: Individual survey responses are not retained because finer-grained data could enable re-identification of participants or organizations.The aggregated results in Table 5 are the complete survey data held by the authors.
  • Limitations on Release: The malware samples are not redistributed; released reports and labels identify each sample by its hash.The authors state that releasable artifacts will become publicly available under a permanent, non-anonymous link upon acceptance.

A Paper Selection

The paper-selection procedure lists the venues searched and the query terms used to identify work on cyber threat intelligence, TTPs, attack analysis, threat sharing, and related topics.

  • Venues: The venue list spans security, systems, networking, data-management, and machine-learning conferences and workshops.The supplied selection record names venues including CCS, IEEE S&P, NDSS, RAID, SIGMOD, and USENIX Security.
  • Search Terms: The first query block searches for cyber threat intelligence extraction, mining, or classification.It also includes terms for TTP, ATT&CK, threat reports, attack patterns, attack techniques, attack graphs, and threat sharing.
  • Search Terms: The query terms include threat report generation, CTI generation, threat mining, and cyber threat investigation.These terms appear alongside threat-sharing and threat-intelligence-sharing expressions.
  • Search Terms: A separate query block searches for cyber threat intelligence as an exact phrase.The supplied record presents this phrase independently from the broader extraction and mining terms.
  • Search Terms: The later query syntax combines extraction, mining, and classification terms with searches for TTP, ATT&CK, threat reports, attack patterns, techniques, and graphs.It also includes threat sharing, threat report generation, threat intelligence, threat detection, threat analysis, and cyber threat investigation.

B Survey Questionnaire and Results

Table 5 presents the survey questionnaire and practitioners’ responses concerning CTI workflows, challenges, and opportunities for AI-driven support.

  • Survey Table: Table 5 contains the survey questionnaire and practitioners’ responses.
  • Survey Table: The table addresses CTI workflows and challenges.
  • Survey Table: The table also covers AI-driven opportunities in CTI work.
Loading 2609.01174v1…