PaperlayerView exact revision on arXiv

Source-linked AI summary

CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence

Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, Nidhi Rastogi

arXiv:2406.07599v3cs.CRcs.AI

TL;DR

LLMs may help process CTI data, but concerns about hallucinations and a lack of practical CTI benchmarks leave their capabilities uncertain. CTIBench evaluates knowledge and applied CTI tasks; GPT-4 leads most tasks, while Gemini-1.5 leads vulnerability severity prediction.

  • Problem

    Existing benchmarks do not evaluate practical CTI tasks comprehensively, leaving LLM reliability and usefulness in this domain an open question.

    Source: §1 · p. 2

  • Method

    CTIBench combines a CTI knowledge dataset with practical tasks for vulnerability mapping, severity prediction, attack-technique extraction, and threat-actor attribution.

    Source: §1 · p. 2

  • Results

    GPT-4 outperforms other models across all tasks except CTI-VSP, where Gemini-1.5 leads; LLAMA3-70B is comparable to Gemini-1.5 and outperforms it on three tasks.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    CTIBench offers researchers a way to examine LLM capabilities and limitations across CTI aspects and support understanding of their practical applications.

    Source: §8 · p. 10

    The evaluation covers only a limited subset of CTI tasks and may not capture the breadth required for comprehensive CTI operations.

    Source: §7 · p. 10

Abstract

from arXiv · show

Cyber threat intelligence (CTI) is crucial in today's cybersecurity landscape, providing essential insights to understand and mitigate the ever-evolving cyber threats. The recent rise of Large Language Models (LLMs) have shown potential in this domain, but concerns about their reliability, accuracy, and hallucinations persist. While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks. To bridge this gap, we introduce CTIBench, a benchmark designed to assess LLMs' performance in CTI applications. CTIBench includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. Our evaluation of several state-of-the-art models on these tasks provides insights into their strengths and weaknesses in CTI contexts, contributing to a better understanding of LLM capabilities in CTI.

1 Introduction

LLMs could help analysts process threat data, but hallucinations and misunderstandings make their reliability in CTI a concern. CTIBench addresses the lack of benchmarks for practical CTI capabilities with tasks spanning knowledge and applied analysis.

1 Introduction

  • CTI lacks benchmarks that objectively measure and compare how effectively LLMs handle practical cybersecurity tasks and understand the domain.Existing general and domain-specific benchmarks do not capture practical cybersecurity aspects, while prior cybersecurity evaluations often focus on memorization rather than broad CTI comprehension and problem-solving.Source: §1 · pp. 1–2
  • CTIBench combines a CTI knowledge quiz with four practical tasks: vulnerability-to-weakness mapping, severity scoring, attack-technique extraction, and threat-actor attribution.The tasks assess understanding, reasoning, and problem-solving across cyber threats, vulnerabilities, and attack patterns.Source: §1 · p. 2
  • The benchmark is intended to support automated security-alert triage and analysis, helping analysts focus on critical threats and reduce response time.The authors describe CTIBench as the first benchmark specifically designed to evaluate LLM comprehension, reasoning, and problem-solving across broad CTI.Source: §1 · p. 2

2 Related Work

LLMs are used across cybersecurity, but evaluation remains incomplete: existing benchmarks are either narrow or do not test practical CTI problem-solving.

Large Language Models in Cybersecurity

  • LLMs have been applied to secure-code generation, vulnerability detection, incident response, program repair, IT operations, and cybersecurity knowledge assistance.Despite these applications, the authors identify a significant gap in cybersecurity-domain evaluation.Source: §2 · p. 2

Evaluating Large Language Models

  • General benchmarks such as GLUE [52], SuperGLUE, MMLU, HELM, and KOLA assess general understanding, not broad practical CTI capabilities.Cybersecurity benchmarks include SECURE [16], Purple Llama CyberSecEval [15], and SecLLMHolmes, which focus on particular domains or insecure-code generation.Source: §2 · p. 3
  • Other cybersecurity evaluations target entity extraction, relationships, summaries, memorization, or information extraction, leaving practical CTI problem-solving unassessed.The authors characterize existing cybersecurity evaluations as either insufficiently comprehensive or too narrow in domain adaptation.Source: §2 · p. 3

3 CTIBench: A Benchmark for Evaluating LLMs in CTI

CTIBench evaluates four cognitive capabilities through a knowledge quiz and practical tasks built from CTI sources, vulnerability records, attack behaviors, and threat reports.

3 CTIBench: A Benchmark for Evaluating LLMs in CTI

  • CTIBench structures its tasks around memorization, understanding, problem-solving, and reasoning about cyber threat reports.The benchmark aims to test whether LLMs can understand, investigate, and analyze those reports.Source: §3 · p. 3

3.1 CTI-MCQ: Cyber Threat Intelligence Multiple Choice Questions

  • CTI-MCQ questions draw on CTI frameworks, standards, vulnerability and attack-pattern resources, and curated public quizzes; manual review corrected wrong answers and removed ambiguous questions.The final dataset contains 2,500 questions: 1,578 from MITRE, 750 from CWE, 40 manually collected, and 32 from standards and frameworks.Source: §3.1 · p. 3

3.2 CTI-RCM: Cyber Threat Intelligence Root Cause Mapping

  • CTI-RCM maps vulnerability descriptions to CWE categories, a nuanced task complicated by more than 900 categories and the need to interpret technical descriptions.The dataset randomly samples 1,000 vulnerabilities reported in 2024 from NVD records with associated CWE mappings.Source: §3.2 · p. 4

3.3 CTI-VSP: Cyber Threat Intelligence Vulnerability Severity Prediction

  • CTI-VSP predicts CVSS v3 vector strings from vulnerability descriptions, evaluating Base metrics rather than Temporal or Environmental metrics.The authors use CVSS v3 because models may not adequately know CVSS 4.0; the dataset contains 1,000 descriptions from 2024 with associated CVSS v3 strings.Source: §3.3 · pp. 4–5

3.4 CTI-ATE: Cyber Threat Intelligence Attack Technique Extraction

  • CTI-ATE extracts MITRE ATT&CK techniques from threat behavior; its dataset includes 30 malware instances reported in 2024 and 30 reported earlier, covering 397 unique techniques.The dataset excludes sub-techniques, and mapping observed behavior to technique IDs is presented as useful for mitigation and targeted security measures.Source: §3.4 · p. 5

3.5 CTI-TAA: Cyber Threat Intelligence Threat Actor Attribution

  • CTI-TAA tests abductive attribution of threat actors from reports that vary in detail, using 50 vendor reports with actor and campaign names replaced by placeholders.Responses were manually checked to account for threat-actor aliases.Source: §3.5 · pp. 5–6

4 Experiments and Results

Five models were evaluated on CTIBench using zero-shot prompts and task-specific metrics. GPT-4 led most tasks, while Gemini-1.5 led CTI-VSP.

4.1 Experimental Settings

  • The evaluation covers ChatGPT-3.5, ChatGPT-4, Gemini-1.5, LLAMA3-70B, and LLAMA3-8B with temperature 0, top_p = 1, and zero-shot cybersecurity-expert prompts.The paper provides a sample prompt for vulnerability root-cause mapping and refers to Appendix C for prompts for all tasks.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • The study uses accuracy for CTI-MCQ and CTI-RCM, MAD for CTI-VSP, and Micro-F1 for CTI-ATE; CTI-TAA reports correct and plausible accuracy.CTI-TAA distinguishes correct, plausible, and incorrect answers; plausible accuracy combines correct and plausible responses.Source: §4.2 · p. 6

4.3 Results Summary

  • GPT-4 outperforms other models across all tasks except CTI-VSP, where Gemini-1.5 leads; LLAMA3-70B is comparable to Gemini-1.5 and beats it on three tasks.LLAMA3-70B struggles on CTI-VSP, while ChatGPT-3.5 generally trails the other models but exceeds LLAMA3-8B, which performs decently on CTI-MCQ.Source: §4.3 · p. 6, Table 1 · p. 7

5 Analysis

Across CTI tasks, model performance varied with question source and CVE-description length; analyses also identified shared MCQ errors, attribution mistakes, and longer GPT-4 responses.

5.1 CTI-MCQ

  • All five evaluated models answered 293 CTI-MCQ questions incorrectly; those questions concerned mitigation plans and adversary tools and techniques.Source: §5.1 · p. 7
  • Larger models had higher CTI-MCQ error correlations, and adding an explicit reasoning prompt did not consistently improve task performance.The authors hypothesize that the task relies more on memorization than reasoning.Source: §5.1 · p. 7
  • 75.65% was ChatGPT-4’s accuracy on CTI-MCQ, and models generally scored better on questions sourced from CWE than from ATT&CK.The analysis describes ATT&CK information as more volatile than CWE information.Source: §5.1 · p. 7

5.2 CTI-RCM

  • CTI-RCM accuracy rose with longer CVE descriptions for all models except LLAMA3-8B, peaking at 54–69 words before declining for the longest descriptions.Four of five models performed slightly worse on 2021 CVEs than on the 2024 dataset.Source: §5.2 · pp. 7–8

5.3 CTI-VSP

  • CVSS prediction improved with longer descriptions until the longest quintiles, while models did better on AV/AC/UI than PR/S/C/I and overestimated more than they underestimated.The reported interpretation is that longer descriptions can add context but also complexity that leads to threat-severity misattribution.Source: §5.3 · pp. 8–9

5.4 CTI-ATE

  • ChatGPT-4 significantly outperformed other models on CTI-ATE, while most models performed slightly better on pre-cutoff samples except Gemini.The before/after-cutoff performance differences were generally insignificant.Source: §5.4 · p. 9

5.5 CTI-TAA

  • Threat-attribution models made nuanced correlations, but smaller models struggled with complex reasoning and ChatGPT-4 sometimes confused actors with shared patterns or histories.ChatGPT-4 predicted OilRig for CHRYSENE, a listed alias, but predicted APT29 for APT41 based partly on shared activity since 2012.Source: §5.5 · p. 9

5.6 Compute Cost

  • GPT-4 generated significantly longer responses than the other models across all evaluated tasks.Source: §5.6 · p. 10

6 Ethical Concerns

The evaluation tasks use publicly available threat information from reputable sources and exclude personal information and sensitive judgments related to social issues.

  • The benchmark tasks use publicly available threat information from sources such as NIST, MITRE, CVE, CWE, and the EU.Source: §6 · p. 10
  • The datasets contain no personal information and make no sensitive judgments about social issues, bias, deception, or discrimination.Source: §6 · p. 10

7 Limitations

The evaluation covers only a subset of CTI tasks and English-language techniques, and does not assess the potential misuse of LLMs for harmful purposes.

  • The study evaluates a limited subset of CTI activities, which may not capture the breadth needed for comprehensive CTI operations.Source: §7 · p. 10
  • The evaluation is restricted to English-language CTI techniques, despite cyber threats frequently spanning multiple languages and regions.Source: §7 · p. 10
  • Misused LLMs could generate compelling but false threat reports that mislead decision-makers, misallocate resources, or prompt inappropriate responses.Benchmarking this potential misuse remains open future work.Source: §7 · p. 10

8 Conclusion

The paper introduces CTIBench to evaluate LLM performance across CTI tasks, offering insights into their capabilities and limitations. The authors aim for it to help researchers understand practical CTI applications and support reliable use and cyberthreat detection and mitigation.

  • CTIBench evaluates LLM performance across various CTI tasks, where their capabilities and limitations remain unclear.Source: §8 · p. 10
  • The evaluation offers insights into LLM knowledge and capabilities across various CTI aspects, as well as their limitations.Source: §8 · p. 10
  • The authors aim for CTIBench to help researchers understand practical LLM applications in CTI and pave the way for reliable use and effective cyberthreat detection and mitigation.Source: §8 · p. 10

Checklist

The checklist marks code, data, instructions, compute details, and new assets as provided, and reports that limitations were discussed. The appendix prompt specifies the format and content requirements for generating CTI multiple-choice questions.

Checklist

  • The checklist marks code, data, and instructions for reproducing the main experimental results as included.Source: Checklist · p. 15
  • The checklist marks the total compute time and resource type as reported.Source: Checklist · p. 15
  • The checklist marks new assets as included and the work's limitations as discussed.Source: Checklist · p. 15

A Prompt for generating CTI-MCQ

  • The CTI-MCQ prompt requires challenging four-option questions for experienced CTI professionals, with precise, context-specific content and TSV output.It also requests technique IDs and names, and platform context where applicable.Source: A Prompt for generating CTI-MCQ · p. 16

B CVSS Base Metric

This block describes CVSS v3.1 base metrics and gives prompt templates for five CTI evaluation tasks.

B CVSS Base Metric

  • Attack Vector, Attack Complexity, and Privileges Required capture attacker proximity, exploit conditions, and required access.The listed values distinguish network, adjacent, local, or physical access; low or high complexity; and none, low, or high privileges.Source: B CVSS Base Metric · pp. 16–17
  • User Interaction records whether exploitation needs user participation, while Scope distinguishes impacts within or beyond the vulnerable component.The listed values are None or Required for User Interaction and Unchanged or Changed for Scope.Source: B CVSS Base Metric · p. 17
  • Confidentiality and Integrity Impact measure information disclosure and data alteration, each with None, Low, and High values.High Confidentiality Impact denotes complete data disclosure; High Integrity Impact denotes extensive or significant modification.Source: B CVSS Base Metric · p. 17
  • Availability Impact measures disruption to the affected component, ranging from none to degraded performance or unavailability.Its values are None, Low, and High.Source: B CVSS Base Metric · pp. 16–17

C.1 CTI-MCQ: Cyber Threat Intelligence Multiple Choice Questions

  • The CTI-MCQ prompt asks the model to choose one of four options and return a single uppercase letter.The prompt frames the model as a CTI cybersecurity expert and requires the final line to contain only the answer letter.Source: §C.1 · pp. 17–18

C.2 CTI-RCM: Cyber Threat Intelligence Root Cause Mapping

  • The CTI-RCM prompt asks the model to map a CVE description to a CWE and briefly justify its choice.It requires the final line to contain only the CWE ID.Source: §4.1 · p. 6, §C.2 · p. 18

C.3 CTI-VSP: Cyber Threat Intelligence Vulnerability Severity Prediction

  • The CTI-VSP prompt requests CVSS v3.1 base-metric values and a final vector string from a CVE description.It specifies valid options for each metric and requires the final line to contain only the vector string.Source: §C.3 · p. 18

C.4 CTI-ATE: Cyber Threat Intelligence Attack Technique Extraction

  • The CTI-ATE prompt asks for MITRE Enterprise attack patterns and technique IDs, with reasoning and main-technique IDs on the final line.The final line excludes subtechnique IDs.Source: §C.4 · pp. 18–19

D CTI-TAA Evaluation

The section describes graph-based evaluation for threat-actor attribution, then reports reasoning-prompt results and shows examples of shared MCQ errors.

D CTI-TAA Evaluation

  • Threat-actor predictions are checked by breadth-first search over aliases, then over aliases and related-group links to classify correct, related, or incorrect responses.A match through aliases counts as correct; a match through the expanded related-group search counts as related.Source: D CTI-TAA Evaluation · p. 19

E CTI-MCQ with reasoning

  • Reasoning prompts substantially increased computational costs but generally did not improve MCQ accuracy; GPT-3.5 notably improved, while LLAMA3-8B declined.Table 4 compares accuracy with and without reasoning and gives approximate generated-token counts in thousands.Source: E CTI-MCQ with reasoning · p. 19, Table 4 · p. 20

F CTI-MCQ Samples

G Compute Cost

The section reports approximate prompt and response token counts by task in Table 6.

G Compute Cost

Loading 2406.07599v3…