CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models

1. Introduction

• CTI reports contain malware, vulnerabilities, attack techniques, and their interactions, whereas IOC-based knowledge covers a limited set of knowledge and has a short lifespan.These richer descriptions support comprehensive cyber threat profiles.Source: §1 · p. 1
• Syntax-parsing and fine-tuning approaches are difficult to adapt to new ontologies and emerging threats because they rely on fixed rules or large labeled CTI datasets.These constraints are especially acute where annotations are scarce for new threats.Source: §1 · p. 1
• CTINEXUS uses LLM in-context learning with ontology-aware prompts and retrieved demonstrations to extract entity-relation triplets without large training datasets or extensive model tuning.Its automatic prompt construction supports swapping ontologies and adapting to new threats and tasks with a few demonstrations.Source: §1 · p. 2
• Hierarchical entity alignment groups entities by type before merging semantically similar mentions, while long-distance relation prediction connects central nodes across disjoint subgraphs.The alignment stages avoid comparing every entity pair with an LLM; relation prediction selects high-degree central nodes for implicit-link inference.Source: §1 · p. 2
• CTINEXUS achieved 87.65% F1 in cybersecurity triplet extraction, 89.94% in coarse-grained entity grouping, 99.80% in fine-grained entity merging, and 90.99% in relation prediction.It outperformed EXTRACTOR by 25.36% in F1-score for cybersecurity triplet extraction and LADDER by 19% in cybersecurity entity extraction.Source: §1 · p. 2

2.1. Cyber Threat Intelligence

• CSKGs represent entities as nodes and relations as edges, providing a holistic threat profile that is easier to visualize and integrate into downstream applications.Threat evolution makes it difficult to maintain one universal, comprehensive ontology.Source: §2.1 · p. 2

2.2. Limitations of Existing Approaches

• Figure 1 compares CSKGs generated for one real-world CTI report; CTINEXUS is described as more integrated and comprehensive than EXTRACTOR, TTPDrill, and LADDER.The other systems tend to produce fragmented subgraphs lacking contextual connections.Source: Figure 1 · p. 3
• Syntax parsing can fail on CTI-specific characters and relies on fixed rules and dictionaries, while fine-tuning demands costly ontology-aligned annotations and is difficult to transfer across ontologies.The methods also require ongoing maintenance of parsing rules and dictionaries.Source: §2.2 · pp. 3–4

2.2.1. Motivating Example.

• In the ransomware example, existing systems produced wrong relation directions, poor or incomplete entity mentions, unaligned entities, and missed links between entities mentioned in different sentences.CTINEXUS’s graph is described as more comprehensive and connected, without large training data, and adaptable to different ontologies.Source: §2.2.1 · p. 4

2.3. Large Language Models (LLMs)

• In-context learning gives an LLM a task instruction, demonstration examples, and a query, enabling adaptation to new tasks with minimal cost through task-specific prompts and examples.Its use for CTI knowledge extraction and CSKG construction remains largely underexplored in the cited discussion.Source: §2.3 · p. 4

Phase 1

• CTINEXUS first extracts entity-relation triplets aligned with the task ontology using a kNN-selected set of demonstrations and an automatically constructed ICL prompt.The prompt includes an instruction, the CTI report query, and demonstrations arranged in a specific order.Source: §3 · p. 4

Phase 2

• Hierarchical entity alignment reduces redundancy by grouping entities by type before merging semantically similar entities within each group.Fine-grained merging uses a predefined similarity threshold.Source: §3 · p. 4

Phase 3

• Long-distance relation prediction selects central nodes from connected subgraphs and uses ICL to infer their implicit relationships with a topic node.The topic node is selected from the central nodes based on importance.Source: §3 · pp. 4–5

4.1. CSKG Ontology

• CTINEXUS uses MALOnt in its current implementation, with 33 entity types and 27 relation types, and can switch to another ontology using a few aligned demonstrations.For a new ontology that is a subset of MALOnt, unrequired entity types can simply be removed.Source: §4.1 · p. 5

4.2. Cybersecurity Triplet Extraction

• Triplet extraction uses a single ICL prompt with task instructions, a query, and top-k demonstrations retrieved by semantic similarity.The retrieved examples are arranged in ascending similarity order, placing the most similar demonstration closest to the query.Source: §4.2 · p. 6

4.3. Hierarchical Entity Alignment

• Hierarchical alignment first assigns entity types and groups entities of the same type, then merges semantically similar entities within each group.IOC protection isolates detected IOC patterns to prevent merging distinct indicators that share a general type.Source: §4.3 · pp. 7–8

4.4. Long-Distance Relation Prediction

• Long-distance relation prediction selects central entities by degree centrality, identifies a topic entity, and uses ICL to infer relations between each central entity and the topic.Selecting central nodes rather than linking every pair of distant entities avoids excessive connections and computational costs.Source: §4.4 · pp. 8–9

• The evaluation asks how CTINEXUS compares with existing methods for CTI knowledge extraction.Source: §5 · p. 9
• It examines how different settings affect cybersecurity triplet extraction, entity alignment, and relation prediction.Source: §5 · p. 9
• The evaluation also asks about end-to-end performance, adaptation to different settings, and efficiency.Source: §5 · p. 9

5.1. Dataset and Metrics

• The evaluation dataset contains 150 recent CTI reports from 10 sources, with annotations covering entities, explicit relations, entity alignment, and implicit relations.The reports were collected from May 2023 onward.Source: §5.1 · p. 9
• Three threat-intelligence PhD students annotated the data, with a third annotator arbitrating conflicts; agreement averaged 0.73 across tasks.Cohen’s kappa was 0.80 for triplet extraction, 0.78 for entity alignment, and 0.61 for relation prediction.Source: §5.1 · p. 9
• Triplets are evaluated by semantic match to gold subject-relation-object triples, while entity alignment requires correct type assignment and merging with the matching gold entity.Active- and passive-voice variants count as equivalent for triplet evaluation.Source: §5.1 · p. 9

5.2. RQ1: How does CTINEXUS compare against existing CTI knowledge extraction methods?

• For LADDER, the authors converted annotations to its word-level format and evaluated named-entity extraction only, since its relation classes are fixed.The training set combined the authors’ data and LADDER’s in a 5:1 ratio; the test set was replaced with theirs.Source: §5.2 · p. 9
• CTINEXUS outperforms EXTRACTOR on every cybersecurity triplet-extraction metric and LADDER on F1-score, precision, and recall.The comparisons are reported in Tables I and II.Source: §5.2 · pp. 9–10, Table I · p. 10, Table II · p. 10

Impact of prompt example numbers

• Two examples are adopted for triplet extraction: more examples increase prompt length, while three-example precision and F1 plateau and four-example precision falls.Each additional example adds an average of 603 tokens; four examples raise recall from 82% to 84% while precision falls from 93% to 89%.Source: §5.3 · p. 10

Impact of prompt example permutations

• kNN selection with examples sorted in ascending query-similarity order outperforms random and descending-order strategies across all reported metrics.The authors associate this result with recency bias and adopt ascending order.Source: §5.3 · pp. 10–11

Impact of backbone models

• GPT-4 improves both recall and precision by over 10% versus GPT-3.5-turbo, while GPT-4, GPT-3.5-turbo, Qwen2.5, and Llama3 are the evaluated backbones.GPT-4 performs best in both precision and recall among the tested models.Source: §5.3 · p. 11

Prompt formulation and design

• End-to-end prompting reduces input token consumption by 98.9% and output token consumption by 97.3% compared with multi-round QA prompting.CTINEXUS extracts all entities and relations in one inference using a uniform prompt template.Source: §5.3 · p. 11
• Compared with a vanilla prompt, the refined prompt improves F1-score by 20.57%, precision by 15.28%, and recall by 24.73%.Its features constrain output format and content, use role-playing, and place instructions before demonstrations.Source: §5.3 · p. 11

5.4.1. Hierarchical Entity Alignment.

• GPT-4 with one demonstration outperforms GPT-3.5 with twelve, and alignment results saturate beyond eight demonstrations.For GPT-4, all three reported metrics slightly decrease when increasing demonstrations from eight to twelve.Source: §5.4.1 · p. 11
• A 0.6 merging threshold has the highest precision among the tested thresholds; 0.4, 0.5, and 0.6 each achieve 100% recall.At 0.7, precision is maintained but recall drops significantly; text-embedding-3-large performs best among the tested embedding models.Source: §5.4.1 · p. 12

5.4.2. Long-Distance Relation Prediction.

• Relation prediction may perform better because it relies more on general linguistic inference than the domain-specific knowledge needed for triplet extraction or entity alignment.The passage presents this as a possible explanation for the relative task performance.Source: §5.4.2 · p. 12
• GPT-4 achieves 100% recall in every tested relation-prediction setting, versus 92%–96% for GPT-3.5.Both models perform suboptimally zero-shot; moving from one to two demonstrations improves results, while three-shot results decline slightly.Source: §5.4.2 · p. 12

5.5. RQ4: How well does CTINEXUS perform in end-to-end CSKG construction?

• On ten reports, end-to-end CTINEXUS achieves 87.80% F1-Score, 81.82% precision, and 94.74% recall.Each pipeline component operates on the preceding component’s output in this evaluation.Source: §5.5 · p. 12

5.6. RQ5: How well does CTINEXUS adapt to different CSKG ontologies?

• On ten STIXnet reports, CTINEXUS achieves 89.7% precision, 81.9% recall, and 85.6% F1-score using the STIX ontology.The evaluation uses 13 selected STIX Domain Objects and 128 manually annotated gold triplets; ontology definitions are injected into the prompt while LLM weights remain frozen.Source: §5.6 · p. 13

5.7. RQ6: What is the efficiency of CTINEXUS?

• Average cost per report is $0.1485 with GPT-4 and $0.0069 with GPT-3.5; GPT-4 costs 20–30 times more in tokens and takes about twice as long.Long-distance relation prediction is the most computationally expensive module; fine-grained merging costs less than $0.30 for the entire experiment.Source: §5.7 · p. 13

6. Discussion

• CTINEXUS requires high-quality, correctly formatted demonstrations and recommends at least 100 labeled samples; imbalanced examples can reduce retrieval diversity and effectiveness.Incorrect or misformatted demonstrations can degrade performance.Source: §6 · p. 13
• Hallucinations can produce incorrect entity-relation triplets, an issue reported as more prevalent in smaller models including GPT-3.5, LLaMA3-70B, and QWen2.5-72B.The authors leave hallucination-detection classifiers and stronger-model verification for future work, while reporting reliable GPT-4 performance.Source: §6 · p. 13
• Constructed knowledge graphs could support STIX conversion, threat-question answering, threat hunting, and investigations, but these downstream applications remain future work.The authors describe exchanging converted knowledge through platforms such as AlienVault OTX and using graphs to suggest investigation steps.Source: §6 · pp. 13–14

CTI services and platforms

• CTI services range from isolated IOC feeds to community-sharing and management platforms, which rely on users to contribute data rather than automated extraction.Examples include PhishTank, AlienVault OTX, MISP, and OpenCTI.Source: §7 · p. 14

Cybersecurity knowledge bases

• Expert-curated resources such as CVE, NVD, and MITRE ATT&CK typically update infrequently, while CTINEXUS could structure textual CTI descriptions in these knowledge bases.Source: §7 · p. 14

LLMs for cybersecurity

• Other cybersecurity LLM work addresses penetration testing, fuzzing, vulnerability detection, patching, malware, and phishing, whereas CTINEXUS targets CTI extraction and graph construction.Source: §7 · p. 14

Other CTI research

• Empirical CTI studies examine vulnerability reproducibility, feed quality, and information inconsistencies, while CTINEXUS designs an automated extraction approach.Source: §7 · p. 14

8. Conclusion and Future Work

• CTINEXUS uses LLM in-context learning for efficient, adaptive CTI extraction and graph construction, and the authors report effective extraction of comprehensive knowledge.They characterize the framework as requiring minimal data and parameter tuning and adapting to various ontologies with minimal annotation.Source: §8 · p. 14
• CTINEXUS is presented as an alternative to costly, slow-to-refresh security-model approaches, supplying structured, current threat knowledge for downstream LLM reasoning across sources.Source: §8 · p. 14
• Integrating visual analytics is proposed as future work to help analysts inspect malicious-activity patterns and relationships and support timely decisions.Source: §8 · p. 14