PaperlayerView exact revision on arXiv

Source-linked AI summary

CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence

Yutong Cheng, Yang Liu, Changze Li, Dawn Song, Peng Gao

arXiv:2510.11974v2cs.CRcs.AI

TL;DR

CTI benchmarks have not evaluated retrieval-augmented LLMs across heterogeneous CTI sources, despite the scale and rapid evolution of this knowledge. CTIConnect builds a benchmark and tests domain-specific retrieval strategies across CTI task categories. The results show that the semantic gap varies by task, and domain-specific strategies outperform general-purpose retrieval approaches.

  • Problem

    No existing benchmark evaluates retrieval-augmented LLMs across the heterogeneous CTI sources analysts use, despite rapid knowledge updates and the limits of parametric memory.

    Source: §1 · pp. 1, 3

  • Method

    CTIConnect integrates five CTI sources into 1,860 expert-verified QA pairs across nine tasks in Entity Linking, Multi-Document Synthesis, and Entity Attribution.

    Source: §1 · p. 3

  • Results

    The cross-source semantic gap varies by task category, and domain-specific retrieval strategies outperform vanilla RAG by up to +35.2% for entity linking, +16.0% for attribution, and +11.3% for synthesis.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Different task categories call for different retrieval strategies, and closing the cross-source gap requires structural interventions rather than generic retrieval improvements.

    Source: §1 · p. 3, §5 · p. 9

Abstract

from arXiv · show

Cyber Threat Intelligence (CTI) is foundational to modern cybersecurity, enabling organizations to proactively defend against evolving threats. However, the sheer volume and heterogeneity of CTI data, spanning structured knowledge bases (CVE, CWE, CAPEC, MITRE ATT&CK) and unstructured threat reports, far exceed the capacity of manual analysis. The strong contextual understanding and reasoning of Large Language Models (LLMs) have driven growing interest in applying them to CTI tasks. Yet no existing benchmark evaluates LLMs in a retrieval-augmented setting with a proper evaluation harness that grants access to the heterogeneous domain knowledge sources analysts rely on in practice. To address this gap, we present CTIConnect, a benchmark for systematically evaluating retrieval-augmented LLMs across the CTI task landscape. We construct a unified evaluation environment integrating five heterogeneous CTI sources into 1,860 expert-verified QA pairs spanning nine tasks across three categories: Entity Linking, Multi-Document Synthesis, and Entity Attribution. Extensive experiments on ten state-of-the-art LLMs reveal that the cross-source semantic gap manifests differently across task categories, demanding fundamentally different retrieval strategies, and that the performance bottleneck shifts between retrieval infrastructure and evidence utilization depending on the task. Our domain-specific strategies further outperform stronger general-purpose retrieval paradigms (retrieve-then-rerank, IRCoT), showing that closing this gap requires structural interventions rather than generic retrieval improvements. These findings hold across all ten LLMs, remain consistent on the full benchmark, and stay stable under temporal splits spanning 2008-2025. Together, they provide actionable guidance for designing scalable retrieval architectures over heterogeneous CTI ecosystems.

1 Introduction

CTI's scale, rapid evolution, and heterogeneous sources make manual analysis difficult and create a need to evaluate retrieval-augmented LLMs. CTIConnect addresses the lack of a benchmark for testing retrieval across this ecosystem.

1 Introduction

  • Existing CTI benchmarks test closed-book knowledge or fine-tuning, not retrieval over the heterogeneous sources used in practice.CTI knowledge changes rapidly, while accumulated domain knowledge exceeds what parametric memory can internalize and factual precision remains critical.Source: §1 · pp. 1, 3
  • The benchmark covers all nine CTI tasks, while existing benchmarks address only subsets.The benchmark's findings are reported across ten LLMs and remain stable under temporal splits spanning 2008–2025.Source: Table I · p. 3, §1 · p. 3
  • CTI spans structured knowledge bases and narrative vendor reports whose differing vocabularies and abstraction levels create cross-source retrieval challenges.For example, a formal ATT&CK technique and an analyst's narrative may describe the same behavior in different terms.Source: §1 · p. 3
  • CTIConnect integrates five knowledge sources into nine tasks across Entity Linking, Multi-Document Synthesis, and Entity Attribution.Its evaluation compares domain-specific retrieval strategies with vanilla RAG baselines.Source: Figure 1 · p. 2, §1 · p. 3
  • Domain-specific strategies improve over vanilla RAG by up to +35.2% for entity linking, +16.0% for attribution, and +11.3% for synthesis.Retrieve-then-rerank and IRCoT recover at most 1–5% of the gap closed by the domain-specific strategies.Source: §1 · p. 3

2 Related Work

Prior CTI work includes closed-book benchmarks, single-report datasets, and efforts to extract or internalize threat knowledge. CTIConnect instead evaluates whether LLMs can connect analysts with heterogeneous CTI sources through retrieval and integration.

CTI Benchmarks

  • CTIBench evaluates four CTI tasks using closed-book models, a small manually curated dataset, and no retrieval augmentation.Its tasks cover only a narrow slice of CTI analysis and are difficult to extend to emerging threats.Source: §2 · p. 4
  • SEvenLLM-Bench covers 28 CTI-related tasks but focuses on single-report extraction and summarization with instruction-tuned models of at most 14B parameters.It excludes frontier LLMs.Source: §2 · p. 4

LLMs for Security

  • Earlier work extracted information from CTI sources, built knowledge graphs, or applied LLMs to downstream operations; CTIConnect evaluates retrieval and integration across sources.The paper frames this as a complementary approach to extracting knowledge or training models to internalize it.Source: §2 · p. 4

3 CTIConnect Design

CTIConnect organizes cross-source CTI operations into three task categories and constructs corresponding QA datasets from authoritative mappings, expert annotations, and vendor reports. A unified pipeline synthesizes and verifies the benchmark pairs.

3.1 Task Design

  • The task design covers structured-to-structured linking, unstructured-to-structured attribution, and unstructured-to-unstructured synthesis.Together, these categories span the cross-source directions over heterogeneous CTI data.Source: §3.1 · p. 4, §3 · p. 4

3.1.1 Entity Linking.

  • Entity linking maps entries between structured CTI sources, where large candidate spaces and subtle terminology distinctions make vocabulary alignment important.The four tasks map CVE→CWE, CWE→CVE, CAPEC→ATT&CK, and CWE→CAPEC.Source: §3.1.1 · pp. 4–5

3.1.2 Entity Attribution.

  • Entity attribution grounds narrative attack or vulnerability descriptions in formal taxonomy entries by decomposing behaviors and retrieving for each.The tasks identify ATT&CK techniques or CWE categories; vulnerability narratives may require inferring root causes from exploitation effects.Source: §3.1.2 · p. 5

3.1.3 Multi-Document Synthesis.

  • Multi-document synthesis aggregates reports about the same entity, addressing inconsistent aliases that create near-miss retrieval distractors.The tasks profile threat actors, trace malware lineage over time, and reconstruct campaign timelines from fragmented accounts.Source: §3.1.3 · p. 5

3.1.4 Structured Cross-Source Mappings.

  • The four entity-linking datasets derive 1,139 QA pairs from official MITRE and NVD mappings across four cross-source directions.The mappings include 200K+ CVE–CWE pairs and 500+ CAPEC–ATT&CK relationships.Source: §3.1.4 · p. 5

3.1.5 Blog-to-Framework Alignments.

  • The Blog-to-Framework dataset yields 380 QA pairs by linking threat-report behavior phrases to ATT&CK techniques or CWE categories through practitioner annotation and expert-verified LLM-assisted mapping.Source: §3.1.5 · p. 5

3.1.6 Report Clusters.

  • The synthesis set aggregates 321 reports from 35 vendors and derives 341 QA pairs from clusters containing reports from at least two vendors.Clustering uses topic identification, alias resolution, and metadata reconciliation.Source: §3.1.6 · pp. 5–6

3.1.7 Construction Pipeline.

  • All data sources use a three-stage pipeline: seed annotation, template-constrained QA synthesis, and multi-layered quality control.Quality control combines an LLM judge, independent practitioner verification, and senior adjudication; the resulting benchmark has 1,860 QA pairs across nine tasks.Source: §3.1.7 · p. 6

4 Experiments

Across nine CTI tasks, experiments show that domain-specific retrieval raises performance but that retrieval gaps and model-capability bottlenecks vary by task. Results are checked on the full benchmark and across temporal splits.

4.1 Experimental Setup

  • The setup compares closed-book, vanilla RAG, and task-specific retrieval across ten LLMs, using F1 for linking and attribution and a validated GPT-4 judge for synthesis.Strategies include Extract-then-Retrieve for linking, CSKG-Guided RAG for synthesis, and Decompose-then-Retrieve for attribution.Source: §4.1 · pp. 6–7

4.2 Overall Performance on CTIConnect

  • On the 691-pair subset, domain-specific retrieval yields 74.3% average performance, with entity linking at 97%, multi-document synthesis at 55.3%, and entity attribution at 58.5%.GPT-5 leads overall at 81.4%, but no model dominates every category.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3.1 Semantic Gap Analysis.

  • Semantic gaps and vanilla-RAG ranks differ by category: gold evidence ranks 4.2 for linking, 6.5 for attribution, and 9.2 for synthesis.The respective cosine-similarity gaps are 0.06, 0.31, and 0.43; synthesis is compounded by aliasing and noisy report chunks.Source: §4.3.1 · p. 8, Figure 5 · p. 8

4.3.2 Evidence Utilization Analysis.

  • With correct evidence, strong-tier models score 89% versus 61% for weak-tier models; with incorrect evidence, GPT-5 scores 42% while LLaMA-3-8B scores 18%.Domain-specific retrieval improves weak-tier models by 36.1% versus 25.2% for strong-tier models.Source: §4.3.2 · p. 8, Figure 6 · p. 8

4.3.3 Model Scaling Analysis.

  • Scaling yields gains of +19.2% for entity attribution, compared with +3.4% for entity linking and +2.7% for synthesis.Retrieval infrastructure is the bottleneck for linking and synthesis, while attribution depends more on model capability; LLaMA-3 gains +29.5% from 8B to 405B.Source: §4.3.3 · pp. 8–9, Table IV · p. 8

4.3.4 Comparison with General-Purpose Retrieval Improvements.

  • Retrieve-then-rerank and IRCoT improve over vanilla RAG by 1–5% on average, below the 5–30% vanilla-to-domain-specific gap.The comparison uses GPT-4o and holds the encoder constant; the authors attribute the larger gains to structural interventions addressing vocabulary and entity-resolution mismatches.Source: §4.3.4 · p. 9, Table V · p. 9

4.3.5 Full-Benchmark Verification.

  • On the full 1,860-pair benchmark, the three tested models retain their ranking and category-difficulty ordering, with absolute scores shifting by only 1–3.5%.The broader set includes harder cases such as rarer cross-reference mappings and TTP-evolution questions on contradictory reports.Source: §4.3.5 · p. 9, Table VI · p. 9

4.3.6 Temporal Robustness.

  • Across temporal halves, performance remains stable, with typical absolute differences within ±2%; multi-document synthesis spans 1.8–3.2 years per QA, and TAP is entirely cross-temporal.The temporal analysis spans 2008–2025.Source: §4.3.6 · p. 9

5 Conclusion

CTIConnect evaluates retrieval-augmented LLMs across heterogeneous, multi-source CTI. Its results show that task categories need different retrieval strategies and face different performance bottlenecks.

5 Conclusion

  • CTIConnect contains 1,860 high-quality QA pairs grounded in authoritative CTI sources and evaluates retrieval-augmented LLMs across heterogeneous, multi-source CTI.The benchmark is built through a unified construction pipeline that preserves discriminative structure across scales.Source: §5 · p. 9
  • The cross-source semantic gap varies by category, requiring different retrieval strategies; general-purpose retrieval improvements close only a small fraction of the vanilla-to-domain-specific gap.The conclusions hold on the full benchmark and under temporal splits spanning 2008–2025.Source: §5 · p. 9
  • Retrieval infrastructure is the bottleneck for some categories, while entity attribution remains gated by model capability; smaller open-source models match frontier models on retrieval-bottlenecked tasks.The paper points toward task-appropriate retrieval routing in LLM-powered security intelligence platforms.Source: §5 · p. 9

A Validation and Robustness Analyses

The appendix section outlines four validation and robustness analyses and points readers to the project page for code and data.

A Validation and Robustness Analyses

A.1 LLM-as-a-Judge Reliability

The GPT-4 judge is evaluated against expert labels, across repeated runs, and for stylistic bias. The reported tests show strong agreement, high consistency, and negligible aggregate bias differential.

A.1 LLM-as-a-Judge Reliability

  • GPT-4 judge agreement with expert consensus is κ=0.85 overall, with task-level κ values of 0.90 for TAP, 0.85 for MLA, and 0.81 for CSC.Three CTI experts scored 115 claims, with inter-human κ=0.93.Source: §A.1 · p. 11
  • Across five runs, the judge gives identical verdicts on 108/115 claims (93.9%), with per-QA score standard deviation of 0.02.Source: §A.1 · p. 11
  • The aggregate stylistic-bias differential is Δ=0.013, and non-GPT models Gemini-2.5-Pro and LLaMA-3-405B show higher per-model bias than GPT-4o.The reported means are +0.021 for GPT-family models and +0.008 for non-GPT models.Source: §A.1 · p. 11, Table VII · p. 11

A.2 Temporal Generalization

CTIConnect tests temporal generalization across data spanning 2008–2025, using cross-temporal synthesis tasks and per-task median splits for entity linking and attribution. Performance remains stable across temporal halves under all three retrieval configurations.

A.2 Temporal Generalization

  • 67% of CSC QAs, 100% of TAP QAs, and 83% of MLA QAs span multiple years, with average spans of 1.8, 3.2, and 3.2 years, respectively.These multi-document synthesis tasks are cross-temporal by construction.Source: §A.2 · p. 11
  • Entity linking and entity attribution tasks were split at each task’s median temporal cutoff and re-evaluated under all three retrieval configurations.The benchmark covers CTI data from 2008 to 2025.Source: §A.2 · p. 11
  • Differences across temporal halves stay within ±2% across all settings, indicating stable generalization.The reported stability covers all three retrieval configurations.Source: §A.2 · p. 11, Table VIII · p. 12

A.3 CSKG Robustness Analysis

The CSKG-Guided RAG strategy remains robust when its offline entity extractor changes, and its graph construction is a one-time cost shared across evaluated models. Its reported construction expense is small relative to repeated answer generation.

A.3 CSKG Robustness Analysis

  • EtR and DtR use the evaluated model itself for query transformation, whereas CSKG-Guided RAG uses an offline graph built once and shared across models.The first two strategies therefore have no external model dependency for that transformation.Source: §A.3 · p. 12
  • 1.6% average degradation followed replacing GPT-4o with GPT-4o-mini as the offline entity extractor for CSKG-Guided RAG.The evaluation used GPT-4o as the answering model across all three multi-document synthesis tasks.Source: §A.3 · p. 12, Table IX · p. 12
  • Retrieval scores reports with BM25 over shared canonical entities and IDF weighting, relying mainly on whether each entity is extracted at least once rather than phrasing differences.This is the stated reason for robustness to extractor capability.Source: §A.3 · p. 12
  • CSKG construction is a one-time offline cost amortized across queries, while answer generation is performed once per query.For 321 reports, construction costs approximately $24 with GPT-4o or $1.4 with GPT-4o-mini; answer generation remains the dominant deployment cost.Source: §A.3 · p. 12

A.4 Magniber Cross-Source Correlation Case Study

The Magniber case study uses CSKG-Guided retrieval to connect three reports across 2017–2023. Their synthesis reveals operational continuity alongside changing targeting and delivery vectors.

A.4 Magniber Cross-Source Correlation Case Study

  • CSKG-Guided RAG surfaces earlier ThreatPost and CrowdStrike reports sharing canonical entities with the March 2023 Google TAG input.The earlier reports are from October 2017 and August 2021.Source: §A.4 · p. 12
  • Linking three reports reveals a six-year Magniber campaign with consistent family identity, shifting targeting and exploitation chains, and persistent operational infrastructure.The reported chain runs from an initial dropper through PrintNightmare to a SmartScreen bypass, while targeting shifts from South Korea to broader Windows users.Source: §A.4 · p. 12
  • Vanilla RAG misses the 2017 and 2021 reports because naming variations and changing exploit terminology push them below gold-report ranking thresholds.None of the three reports alone establishes the campaign continuity found by cross-report synthesis.Source: §A.4 · p. 12

Related papers

Loading 2510.11974v2…