Source-linked AI summary
EXTRACTOR: Extracting Attack Behavior from Threat Reports
Kiavash Satvat, Rigel Gjomemo, V. N. Venkatakrishnan
TL;DR
CTI reports contain important attack knowledge that is difficult to use because it is embedded in verbose, complex natural language. EXTRACTOR transforms that text through normalization, resolution, summarization, and graph generation into provenance graphs, which successfully support threat detection, while remaining limited by audit-log granularity and natural-language extraction loss.
Problem
CTI reports contain valuable attack knowledge, but their verbosity and linguistic complexity make comprehensive extraction of attack behavior difficult.
Method
EXTRACTOR applies normalization, resolution, summarization, and graph generation to transform complex CTI text into provenance graphs.
Results
EXTRACTOR-generated graphs successfully supported threat detection, with all evaluated scenarios exceeding the detection threshold and graph similarities of 0.91 for Microsoft and 0.85 for TrendMicro.
Takeaways & Limitations
The extracted graphs matched manually drawn expert graphs and were successfully used for threat detection.
Takeaways & Limitations
The approach cannot detect attacks whose relevant information is absent from audit logs and cannot extract behaviors represented only in figures or charts.
Abstract
from arXiv · showhide
The knowledge on attacks contained in Cyber Threat Intelligence (CTI) reports is very important to effectively identify and quickly respond to cyber threats. However, this knowledge is often embedded in large amounts of text, and therefore difficult to use effectively. To address this challenge, we propose a novel approach and tool called EXTRACTOR that allows precise automatic extraction of concise attack behaviors from CTI reports. EXTRACTOR makes no strong assumptions about the text and is capable of extracting attack behaviors as provenance graphs from unstructured text. We evaluate EXTRACTOR using real-world incident reports from various sources as well as reports of DARPA adversarial engagements that involve several attack campaigns on various OS platforms of Windows, Linux, and FreeBSD. Our evaluation results show that EXTRACTOR can extract concise provenance graphs from CTI reports and show that these graphs can successfully be used by cyber-analytics tools in threat-hunting.
I. INTRODUCTION
CTI reports contain valuable attack knowledge, but extracting a concise, relationship-rich account is difficult because the text is verbose and linguistically complex. EXTRACTOR addresses this by transforming unstructured reports into provenance graphs that capture attack actions, artifacts, ordering, and information flow.
- CTI reports describe attack sequences, system effects, and Indicators of Compromise that support cyber operations and response.
- Earlier extraction approaches identify IOCs or threat actions but do not capture their relationships, limiting a comprehensive view of attack behavior.
- Verbose reports contain substantial irrelevant material, so only a small portion may describe attack behavior observable in system audit logs.
- CTI reports use complex language and structures that challenge approaches assuming simple syntax, fixed grammatical patterns, or stable subject–verb–object relations.
- EXTRACTOR simplifies and summarizes complex text, applies Semantic Role Labeling, and constructs graphs representing attack steps, artifacts, causal dependencies, and information flow.
- The system aims to produce actionable intelligence for detection and threat hunting while processing large numbers of CTI reports, and uses text simplification to condense reports without losing useful information.
II. PROBLEM AND BACKGROUND
Extracting actionable provenance graphs from CTI reports requires handling unstructured, verbose, technically complex text while preserving relationships among attack entities and actions.
- Background: Provenance graphs represent attack entities as nodes and actions between them as edges, using names observable in audit logs.They are intended to connect CTI knowledge with threat-hunting activity.
- Challenges: CTI reports are verbose, with useful attack behavior embedded among irrelevant material; one DustySky report contained only 11 relevant sentences across 42 pages.The relevant sentences describe behavior observable in audit logs.
- Challenges: CTI language challenges NLP tools through long or irregular sentences, ellipsis, pronouns, and other structures that obscure actions and their subjects.The domain often packs multiple actions into sentences and uses omitted subjects for action sequences.
- NLP Background: Part-of-speech tagging and dependency parsing can produce incorrect tags and relations on complex technical sentences, while dependency tags do not capture semantics such as temporality or modality.The cited examples include incorrect subject, object, and proper-noun assignments.
III. APPROACH
EXTRACTOR transforms complex CTI text into canonical, explicit, and reduced sentences before generating a provenance graph that captures attack-event relationships.
- Architecture: EXTRACTOR uses four components: Normalization, Resolution, Summarization, and Graph Generation.Normalization and Resolution simplify and disambiguate text; Summarization removes irrelevant behavior; Graph Generation builds the final graph.
- Normalization: Normalization tokenizes long CTI sentences, homogenizes terminology, and converts passive constructions into active form to make individual actions easier to identify.Its tokenizer uses new lines, bullets, enumerations, and headers in addition to classic punctuation.
- Normalization: The transformation examples normalize malware statements by resolving references and converting actions into audit-log-oriented forms such as stores and creates.The examples include Authorization.exe, logged keystrokes, and copies placed in specified locations.
- Normalization: Homogenization maps cybersecurity synonyms to common entities and system-call verbs using CTI noun and system-call dictionaries.For example, C2, C&C, and Command and Control map to IP:.*, while action synonyms map to corresponding system calls.
- Normalization: Passive-to-active conversion uses POS and dependency patterns to identify agents and subjects, including cases where the agent is implicit.The resulting sentences are shorter, active, and likely to express one action each.
B. Resolution
Resolution makes implicit entities explicit so subsequent processing can preserve attack narrative relationships and produce less ambiguous provenance graphs.
- Resolution: Resolution reconciles implicit references with their actual referents because audit logs contain explicitly named entities.It also reduces ambiguity and improves the accuracy of subsequent processing.
- Ellipsis Subject Resolution: The Ellipsis Subject Resolver detects missing subjects, builds candidate subjects from preceding sentences, and favors the closest candidate.It uses POS and dependency parsing together with the system-calls dictionary.
- Pronoun Resolution: Pronoun resolution adapts NeuralCoref and resolves references such as it and itself to Authorization.exe after earlier processing steps.The examples are shown in Figure 4.
- Entity Resolution: Entity resolution substitutes or removes redundant noun and verb phrases when they refer to actionable entities or actions likely to appear in audit logs.It combines POS and dependency parsing with CTI dictionaries and common-phrase knowledge.
- Resolution: After Resolution, sentences contain explicit subjects, objects, and verbs, while Entity Resolution has reduced the text somewhat before the main summarization step.The largest text reduction occurs after Resolution.
C. Text Summarization
EXTRACTOR’s text summarization reduces CTI verbosity by retaining sentences and words that describe observable attack behavior. It combines sentence classification, word-level tagging, and linguistic processing to preserve detection-relevant information.
- Sentence Verbosity: Sentence verbosity separates productive behavioral sentences from non-productive technical context that lacks observable actions.Productive sentences describe malware behavior useful for detection, whereas non-productive sentences provide complementary context.
- Word Verbosity: Word verbosity removes adverbial and adjectival constructs that do not contribute to behavior descriptions.Examples include “However,” “could,” and “also.”
- Architecture: The two-step summarizer uses a BERT classifier for sentence verbosity and a BiLSTM network for word verbosity.BERT provides contextual word representations, while BiLSTM tags sentence components such as Agent, Patient, and Action.
- Component Preservation: The summarizer preserves important objects even when removing components such as temporal markers or other unnecessary sentence elements.The System Entity Extractor helps prevent removal of detection-relevant entities.
- C. Text Summarization: EXTRACTOR’s summarizer removes superfluous report text while retaining sentences describing attack behavior observable in audit logs.This targets both sentence verbosity and word verbosity.
D. Graph Extraction
EXTRACTOR converts simplified CTI text into provenance graphs by using semantic roles and entity filtering to resolve relationships among attack entities. SRL identifies candidate roles, while SEE removes non-system entities before graph construction.
- Graph Output: The resulting provenance graph represents system entities as nodes and system calls connecting them as edges.The graph shows entities participating in and affected by the attack.
- D. Graph Extraction: EXTRACTOR uses Semantic Role Labeling and rules to extract causality relations and information-flow directions from simplified text.This avoids ambiguous graphs caused by multiple roles and relationships in one sentence.
- Semantic Role Labeling: SRL identifies agents, patients, and actions by assigning semantic roles to verb-related arguments in active and passive sentences.ARG0 denotes the agent and ARG1 the patient.
- Entity Filtering: EXTRACTOR treats SRL arguments as potential attack subjects or objects, then uses SEE to prune arguments that are not system entities.SEE recognizes entities such as files, processes, IP addresses, and registry keys.
2) Graph Builder (GB):
The Graph Builder converts pruned semantic-role outputs into provenance graphs by merging entity arguments, creating node-edge-node triples, and orienting edges according to system-call flow.
- 2) Graph Builder (GB):: The Graph Builder merges SRL arguments with identical text and uses SEE to prune words that are not system entities.This produces the nodes used in the final graph.
- Triple Construction: For each sentence containing a verb role and two nodes, GB generates node-edge-node triples.The verb role represents a system-call connector.
- Edge Direction: GB determines edge direction with a system-call dictionary that maps calls to information-flow directions.For example, send flows from subject to object, whereas recv flows from object to subject.
- Graph Output: The final graph connects attack entities with system-call edges and represents causality and information flow.The output is a provenance graph showing entities participating in and affected by the attack.
V. EVALUATION
EXTRACTOR was evaluated on public CTI reports and DARPA Transparent Computing campaigns using graph-quality measures and threat-hunting tests. Its graphs were comparable to manual graphs, achieved an average F-1 score of 93%, and supported successful detection despite some extraction errors.
- V. EVALUATION: EXTRACTOR’s evaluation covers public incident reports, DARPA Transparent Computing attack descriptions, and a large-scale CTI-report experiment.The experiments use reports with distinct writing styles.
- Evaluation Design: The first two experiments measure attack-behavior capture with precision, recall, and F1-score, then test the graphs in POIROT threat hunting.False-positive and false-negative edges are also measured.
- Public CTI Reports: EXTRACTOR-generated graphs had node and edge counts comparable to manually constructed graphs in public CTI reports.Differences arose from wildcard abstraction and details selected automatically but omitted manually.
- Threat Hunting: All public-report scenarios exceeded the detection threshold t ≈0.3, and POIROT detected the attacks successfully.The authors conclude that automatically generated graphs were as useful as human-generated graphs for threat detection.
- Graph Quality: 93% average F-1 score was achieved when extracted graph edges were compared with attack activities stated in the reports.Residual false positives and false negatives were associated with inverted edges, unresolved ambiguities, and missed entities.
- Robustness: POIROT detected attacks even when EXTRACTOR graphs contained false-positive edges.The evaluation therefore demonstrates detection robustness in the tested scenarios, not perfect graph extraction.
B. Evaluation on the DARPA Transparent Computing Dataset.
EXTRACTOR generated attack-behavior graphs from DARPA Transparent Computing reports and used them for successful threat detection across evaluated scenarios.
- EXTRACTOR processed natural-language attack descriptions from DARPA Transparent Computing campaigns involving client, mail, web, and SSH servers.The campaigns ran on infrastructure defended by blue teams over several weeks.
- The generated graphs were compared with attacker-produced manual graphs as ground truth.
- In all scenarios, POIROT detected the attack because the score exceeded the threshold t ≈0.3.
- EXTRACTOR performed better on DARPA reports than on public CTI reports because the DARPA reports were simpler and produced fewer false positives.Most false negatives resulted from failure to derive relations from sentences.
- POIROT raised no false signals on 12GB of benign audit logs from Windows, Linux, and FreeBSD.This indicates that the extracted graphs were concise enough for threat detection in that benign dataset.
C. Large Scale Experiment
A large-scale evaluation used unstructured reports from Microsoft Security Intelligence and TrendMicro, addressing the absence of direct ground truth through attack-solution graph comparison and manual checks.
- The evaluation processed unstructured CTI reports from Microsoft Security Intelligence and TrendMicro Threat Encyclopedia.
- The absence of ground-truth graphs was the main challenge in evaluating these CTI sources.
- EXTRACTOR compared an attack-description graph with the inverted solution graph using Maximum Common Subgraph similarity.
- The experiment measured report size, sentence counts before and after summarization, and graph characteristics across evaluated reports.
- Manual spot-checks of 50 randomly chosen reports produced precision 0.88, recall 0.93, and F1-score 0.90.Assessments were discussed and agreed upon collectively; false positives and negatives remained minimal relative to report complexity.
D. Fine Grained Performance Evaluation
Fine-grained evaluation examined EXTRACTOR’s summarization, module contributions, and error behavior, while identifying limitations in graph-based coverage and dictionary-dependent processing.
- Text Summarization: BERT outperformed CNN and LSTM on the sentence-verbosity removal task.
- Ablation Study: The ablation study measured each module’s contribution by comparing graphs without that module against the full baseline using MCS similarity.
- False Positives and False Negatives: EXTRACTOR graphs can contain extraneous or missing nodes and edges because natural-language complexity makes some precision loss inevitable.Approximate matching enables threat identification despite this extra or missing information.
- Limitations: The graph-based approach does not apply to timing or side-channel attacks because audit logs lack the required detection granularity.
- Limitations: EXTRACTOR extracts attack behaviors described in natural language but cannot resolve behaviors represented in figures or charts.
- Limitations: Dictionary-based modules may miss words absent from the selected dictionaries, leaving room for further improvement.The paper also identifies NER and vulnerability-report processing as possible extensions.
VII. RELATED WORK
Related work includes audit-log provenance graphs, CTI extraction systems, malware-analysis graphs, and vulnerability-report NLP; EXTRACTOR distinguishes itself by building actionable attack-behavior provenance graphs from natural-language CTI.
- Provenance Graph Analytics: Prior provenance-graph studies use audit logs for attack reconstruction, forensic analysis, and threat hunting, while EXTRACTOR can provide those approaches with attack-focused context.
- Provenance Graph Analytics: VirusTotal Graph represents a high-level malware-sample view, whereas EXTRACTOR converts publicly available CTI reports into actionable knowledge.
- NLP and Threat Information Extraction: iACE, ChainSmith, TTPDrill, and related systems extract IOCs, threat actions, campaign stages, or TTP chains rather than EXTRACTOR’s full attack-behavior provenance graph.
- NLP and Threat Information Extraction: Other related systems address vulnerability reports, malware features, web policies, permissions, sensitive inputs, or binary-code signatures rather than the actual behavior of attacks in CTI.
- EXTRACTOR was evaluated on threat reports and real-world attack scenarios, producing graphs that matched expert-drawn graphs and supported threat detection.
X. APPENDIX
The appendix includes a noun-dictionary example, malware-report characteristics, and generalized graphs that retain IOCs while marking unknown system entities.
- Table XIII presents sample noun phrases alongside their corresponding synonyms in a noun dictionary.
- Table XIV summarizes malware report details and characteristics.
- Figure 10 shows graphs after generalization, retaining IOCs and marking unknown system entities with asterisks.