EXTRACTOR: Extracting Attack Behavior from Threat Reports

• CTI reports describe attack sequences, system effects, and Indicators of Compromise that support cyber operations and response.Source: §I · p. 1
• Earlier extraction approaches identify IOCs or threat actions but do not capture their relationships, limiting a comprehensive view of attack behavior.Source: §I · p. 1
• Verbose reports contain substantial irrelevant material, so only a small portion may describe attack behavior observable in system audit logs.Source: §I · p. 1
• CTI reports use complex language and structures that challenge approaches assuming simple syntax, fixed grammatical patterns, or stable subject–verb–object relations.Source: §I · p. 1
• EXTRACTOR simplifies and summarizes complex text, applies Semantic Role Labeling, and constructs graphs representing attack steps, artifacts, causal dependencies, and information flow.Source: §I · pp. 1–2
• The system aims to produce actionable intelligence for detection and threat hunting while processing large numbers of CTI reports, and uses text simplification to condense reports without losing useful information.Source: §I · p. 2

Background

• Provenance graphs represent attack entities as nodes and actions between them as edges, using names observable in audit logs.They are intended to connect CTI knowledge with threat-hunting activity.Source: A. Problem Description · p. 2

Challenges

• CTI reports are verbose, with useful attack behavior embedded among irrelevant material; one DustySky report contained only 11 relevant sentences across 42 pages.The relevant sentences describe behavior observable in audit logs.Source: B. Challenges · p. 3
• CTI language challenges NLP tools through long or irregular sentences, ellipsis, pronouns, and other structures that obscure actions and their subjects.The domain often packs multiple actions into sentences and uses omitted subjects for action sequences.Source: B. Challenges · p. 3

NLP Background

• Part-of-speech tagging and dependency parsing can produce incorrect tags and relations on complex technical sentences, while dependency tags do not capture semantics such as temporality or modality.The cited examples include incorrect subject, object, and proper-noun assignments.Source: Figure 2 · p. 3, C. NLP Background · p. 4

Architecture

• EXTRACTOR uses four components: Normalization, Resolution, Summarization, and Graph Generation.Normalization and Resolution simplify and disambiguate text; Summarization removes irrelevant behavior; Graph Generation builds the final graph.Source: §III · p. 4

Normalization

• Normalization tokenizes long CTI sentences, homogenizes terminology, and converts passive constructions into active form to make individual actions easier to identify.Its tokenizer uses new lines, bullets, enumerations, and headers in addition to classic punctuation.Source: A. Normalization · pp. 4–5
• The transformation examples normalize malware statements by resolving references and converting actions into audit-log-oriented forms such as stores and creates.The examples include Authorization.exe, logged keystrokes, and copies placed in specified locations.Source: Figure 4 · p. 5
• Homogenization maps cybersecurity synonyms to common entities and system-call verbs using CTI noun and system-call dictionaries.For example, C2, C&C, and Command and Control map to IP:.*, while action synonyms map to corresponding system calls.Source: A. Normalization · p. 5
• Passive-to-active conversion uses POS and dependency patterns to identify agents and subjects, including cases where the agent is implicit.The resulting sentences are shorter, active, and likely to express one action each.Source: A. Normalization · p. 5

Resolution

• Resolution reconciles implicit references with their actual referents because audit logs contain explicitly named entities.It also reduces ambiguity and improves the accuracy of subsequent processing.Source: B. Resolution · p. 6

Ellipsis Subject Resolution

• The Ellipsis Subject Resolver detects missing subjects, builds candidate subjects from preceding sentences, and favors the closest candidate.It uses POS and dependency parsing together with the system-calls dictionary.Source: B. Resolution · p. 6

Pronoun Resolution

• Pronoun resolution adapts NeuralCoref and resolves references such as it and itself to Authorization.exe after earlier processing steps.The examples are shown in Figure 4.Source: B. Resolution · p. 6

Entity Resolution

• Entity resolution substitutes or removes redundant noun and verb phrases when they refer to actionable entities or actions likely to appear in audit logs.It combines POS and dependency parsing with CTI dictionaries and common-phrase knowledge.Source: B. Resolution · p. 6

Resolution

• After Resolution, sentences contain explicit subjects, objects, and verbs, while Entity Resolution has reduced the text somewhat before the main summarization step.The largest text reduction occurs after Resolution.Source: B. Resolution · p. 6

Sentence Verbosity

• Sentence verbosity separates productive behavioral sentences from non-productive technical context that lacks observable actions.Productive sentences describe malware behavior useful for detection, whereas non-productive sentences provide complementary context.Source: C. Text Summarization · p. 7

Word Verbosity

• Word verbosity removes adverbial and adjectival constructs that do not contribute to behavior descriptions.Examples include “However,” “could,” and “also.”Source: C. Text Summarization · p. 7

Architecture

• The two-step summarizer uses a BERT classifier for sentence verbosity and a BiLSTM network for word verbosity.BERT provides contextual word representations, while BiLSTM tags sentence components such as Agent, Patient, and Action.Source: C. Text Summarization · p. 7

Component Preservation

• The summarizer preserves important objects even when removing components such as temporal markers or other unnecessary sentence elements.The System Entity Extractor helps prevent removal of detection-relevant entities.Source: C. Text Summarization · pp. 7–8

C. Text Summarization

• EXTRACTOR’s summarizer removes superfluous report text while retaining sentences describing attack behavior observable in audit logs.This targets both sentence verbosity and word verbosity.Source: C. Text Summarization · pp. 6–8

Graph Output

• The resulting provenance graph represents system entities as nodes and system calls connecting them as edges.The graph shows entities participating in and affected by the attack.Source: D. Graph Extraction · p. 8

D. Graph Extraction

• EXTRACTOR uses Semantic Role Labeling and rules to extract causality relations and information-flow directions from simplified text.This avoids ambiguous graphs caused by multiple roles and relationships in one sentence.Source: D. Graph Extraction · p. 8

Semantic Role Labeling

• SRL identifies agents, patients, and actions by assigning semantic roles to verb-related arguments in active and passive sentences.ARG0 denotes the agent and ARG1 the patient.Source: 1) Semantic Role Labeling (SRL) · p. 8

Entity Filtering

• EXTRACTOR treats SRL arguments as potential attack subjects or objects, then uses SEE to prune arguments that are not system entities.SEE recognizes entities such as files, processes, IP addresses, and registry keys.Source: 1) Semantic Role Labeling (SRL) · p. 8

2) Graph Builder (GB):

• The Graph Builder merges SRL arguments with identical text and uses SEE to prune words that are not system entities.This produces the nodes used in the final graph.Source: 2) Graph Builder (GB) · p. 8

Triple Construction

• For each sentence containing a verb role and two nodes, GB generates node-edge-node triples.The verb role represents a system-call connector.Source: 2) Graph Builder (GB) · p. 8

Edge Direction

• GB determines edge direction with a system-call dictionary that maps calls to information-flow directions.For example, send flows from subject to object, whereas recv flows from object to subject.Source: 2) Graph Builder (GB) · pp. 8–9

Graph Output

• The final graph connects attack entities with system-call edges and represents causality and information flow.The output is a provenance graph showing entities participating in and affected by the attack.Source: 2) Graph Builder (GB) · p. 9

V. EVALUATION

• EXTRACTOR’s evaluation covers public incident reports, DARPA Transparent Computing attack descriptions, and a large-scale CTI-report experiment.The experiments use reports with distinct writing styles.Source: §V · p. 10

Evaluation Design

• The first two experiments measure attack-behavior capture with precision, recall, and F1-score, then test the graphs in POIROT threat hunting.False-positive and false-negative edges are also measured.Source: §V · p. 10

Public CTI Reports

• EXTRACTOR-generated graphs had node and edge counts comparable to manually constructed graphs in public CTI reports.Differences arose from wildcard abstraction and details selected automatically but omitted manually.Source: A. Evaluation on Public CTI reports · p. 10

Threat Hunting

• All public-report scenarios exceeded the detection threshold t ≈0.3, and POIROT detected the attacks successfully.The authors conclude that automatically generated graphs were as useful as human-generated graphs for threat detection.Source: A. Evaluation on Public CTI reports · p. 11

Graph Quality

• 93% average F-1 score was achieved when extracted graph edges were compared with attack activities stated in the reports.Residual false positives and false negatives were associated with inverted edges, unresolved ambiguities, and missed entities.Source: A. Evaluation on Public CTI reports · p. 11

Robustness

• POIROT detected attacks even when EXTRACTOR graphs contained false-positive edges.The evaluation therefore demonstrates detection robustness in the tested scenarios, not perfect graph extraction.Source: A. Evaluation on Public CTI reports · p. 11

• EXTRACTOR processed natural-language attack descriptions from DARPA Transparent Computing campaigns involving client, mail, web, and SSH servers.The campaigns ran on infrastructure defended by blue teams over several weeks.Source: B. Evaluation on the DARPA Transparent… · p. 11
• The generated graphs were compared with attacker-produced manual graphs as ground truth.Source: B. Evaluation on the DARPA Transparent… · p. 11
• In all scenarios, POIROT detected the attack because the score exceeded the threshold t ≈0.3.Source: B. Evaluation on the DARPA Transparent… · p. 12
• EXTRACTOR performed better on DARPA reports than on public CTI reports because the DARPA reports were simpler and produced fewer false positives.Most false negatives resulted from failure to derive relations from sentences.Source: B. Evaluation on the DARPA Transparent… · p. 12
• POIROT raised no false signals on 12GB of benign audit logs from Windows, Linux, and FreeBSD.This indicates that the extracted graphs were concise enough for threat detection in that benign dataset.Source: B. Evaluation on the DARPA Transparent… · p. 12

• The evaluation processed unstructured CTI reports from Microsoft Security Intelligence and TrendMicro Threat Encyclopedia.Source: C. Large Scale Experiment · p. 12
• The absence of ground-truth graphs was the main challenge in evaluating these CTI sources.Source: C. Large Scale Experiment · p. 12
• EXTRACTOR compared an attack-description graph with the inverted solution graph using Maximum Common Subgraph similarity.Source: C. Large Scale Experiment · p. 13
• The experiment measured report size, sentence counts before and after summarization, and graph characteristics across evaluated reports.Source: C. Large Scale Experiment · p. 13, Table VII · p. 13
• Manual spot-checks of 50 randomly chosen reports produced precision 0.88, recall 0.93, and F1-score 0.90.Assessments were discussed and agreed upon collectively; false positives and negatives remained minimal relative to report complexity.Source: C. Large Scale Experiment · p. 13

Text Summarization

• BERT outperformed CNN and LSTM on the sentence-verbosity removal task.Source: D. Fine Grained Performance Evaluation · p. 13

Ablation Study

• The ablation study measured each module’s contribution by comparing graphs without that module against the full baseline using MCS similarity.Source: D. Fine Grained Performance Evaluation · p. 14

False Positives and False Negatives

• EXTRACTOR graphs can contain extraneous or missing nodes and edges because natural-language complexity makes some precision loss inevitable.Approximate matching enables threat identification despite this extra or missing information.Source: §VI · p. 14

Limitations

• The graph-based approach does not apply to timing or side-channel attacks because audit logs lack the required detection granularity.Source: §VI · pp. 14–15
• EXTRACTOR extracts attack behaviors described in natural language but cannot resolve behaviors represented in figures or charts.Source: §VI · p. 15
• Dictionary-based modules may miss words absent from the selected dictionaries, leaving room for further improvement.The paper also identifies NER and vulnerability-report processing as possible extensions.Source: §VI · p. 15

Provenance Graph Analytics

• Prior provenance-graph studies use audit logs for attack reconstruction, forensic analysis, and threat hunting, while EXTRACTOR can provide those approaches with attack-focused context.Source: §VII · p. 15
• VirusTotal Graph represents a high-level malware-sample view, whereas EXTRACTOR converts publicly available CTI reports into actionable knowledge.Source: §VII · p. 15

NLP and Threat Information Extraction

• iACE, ChainSmith, TTPDrill, and related systems extract IOCs, threat actions, campaign stages, or TTP chains rather than EXTRACTOR’s full attack-behavior provenance graph.Source: §VII · p. 15, §VIII · p. 15
• Other related systems address vulnerability reports, malware features, web policies, permissions, sensitive inputs, or binary-code signatures rather than the actual behavior of attacks in CTI.Source: §VII · p. 15

• EXTRACTOR was evaluated on threat reports and real-world attack scenarios, producing graphs that matched expert-drawn graphs and supported threat detection.Source: §VIII · p. 15

• Table XIII presents sample noun phrases alongside their corresponding synonyms in a noun dictionary.Source: Table XIII · p. 17
• Table XIV summarizes malware report details and characteristics.Source: Table XIV · p. 18
• Figure 10 shows graphs after generalization, retaining IOCs and marking unknown system entities with asterisks.Source: Figure 10 · p. 18