Benchmarking LLMs for Threat Level Determination

Han Wang; Murathan Kurfalı; Alfonso Iacovazzi

Source-linked AI summary

Benchmarking LLMs for Threat Level Determination

Han Wang, Murathan Kurfalı, Alfonso Iacovazzi

arXiv:2609.07582v1Sep 7, 2026cs.CR

TL;DR

The paper addresses limited evidence about whether LLMs can reliably determine threat levels in heterogeneous cyber threat intelligence. It builds a curated MISP-based benchmark and evaluates zero-shot and fine-tuned models, finding weak zero-shot performance and substantially better but still operationally insufficient fine-tuned results.

Problem
Evidence is limited on LLM performance for threat-level determination in heterogeneous CTI, despite the operational importance of reducing manual assignment.
Method
The study constructs a curated MISP OSINT dataset, designs a tailored prompt, compares zero-shot models, and applies supervised fine-tuning with alternative input representations.
Results
Fine-tuning raises macro-F1 by 20–35 points relative to zero-shot baselines, with Llama-3.3-70B reaching 0.574 macro-F1 and 0.704 accuracy using textual input.
Takeaways & Limitations
Task-specific adaptation is essential for improving LLM performance on threat-level determination in CTI.
Takeaways & Limitations
Overall performance remains insufficient for reliable operational deployment.

Abstract

from arXiv · show

The fast progress of large language models (LLMs) opens new opportunities in the management of cyber threat intelligence, but their reliability for operational tasks remains unclear. In this work, we benchmark LLMs on the task of threat level determination. First, we construct a curated dataset derived from publicly available MISP OSINT feeds. Next, we design a tailored prompt to systematically compare eight different LLMs under zero-shot conditions. Finally, we apply supervised fine-tuning on each model and perform a comparative analysis between baseline and fine-tuned versions. Our results show that zero-shot models achieve weak performance, with limited ability to correctly assign threat levels. Fine-tuned models, however, demonstrate substantial improvements, reaching F1 scores between 0.40 and 0.58 depending on the base architecture. Despite this progress, the performance is still low for practical deployment, highlighting the need for additional research on data quality, model adaptation, and domain-specific tuning.

I. INTRODUCTION

The paper addresses the underexplored automation of threat-level determination in cyber threat intelligence, where manual processes are costly and error-prone. It introduces a MISP-derived benchmark and compares zero-shot, fine-tuned, and input-representation configurations for LLMs.

Manual threat-level assignment is time-intensive and susceptible to human error, motivating automated CTI processing and analysis.
Existing LLM research had not examined threat-level determination for individual threat incidents or events.
The benchmark dataset is derived from MISP OSINT feeds and processed through simplification and normalization to improve consistency, reduce noise, and facilitate interpretability.
The evaluation uses zero-shot prompting followed by supervised fine-tuning to compare inherent model knowledge with task-specific adaptability.
The study claims to be the first assessment of LLM performance on threat-level determination and aims to reduce reliance on manual annotation.
The experiments compare original JSON with more concise structured textual representations and find that both fine-tuning and input representation substantially affect outcomes.

II. RELATED WORKS

Prior CTI research applies language models to extraction, knowledge-graph construction, retrieval-augmented analysis, vulnerability scoring, and benchmarked intelligence tasks. These efforts do not explicitly evaluate the broader contextual judgment required for threat-level determination.

BERT-based models have been used to extract tactics, techniques, and procedures from unstructured cyber threat reports, outperforming a TF-IDF and Linear Regression baseline.
Other studies use LLMs to extract attack-level intelligence, construct knowledge graphs, and support real-time threat-intelligence analysis through retrieval-augmented generation.
LLMs have also been evaluated for generating CVSS scores for newly disclosed vulnerabilities using prompt engineering, embeddings, and machine-learning approaches.
CTI benchmarks such as CTIBench organize tasks around threat identification, actor attribution, and attack-methodology analysis.
Unlike CVSS scoring and existing CTI benchmark tasks, threat-level determination integrates heterogeneous indicators to assess overall threat seriousness.
The paper positions its benchmark as complementing existing evaluations by targeting an underexplored operational problem at the intersection of situational awareness and decision support.

III. BENCHMARK DESIGN

The benchmark is designed to automate CTI processing and analysis using a structured, reproducible corpus of threat events from publicly available MISP OSINT feeds. Expert-assigned threat levels provide labels intended to reflect real-world usage.

The dataset is intended to be a structured and reproducible corpus of threat-intelligence events derived from MISP data feeds.
The benchmark targets automation of CTI processing and analysis to reduce human intervention while accelerating and strengthening CTI management.
Raw CTI data was collected from CIRCL and Botvrij.eu MISP OSINT feeds because they provide manually assigned threat levels alongside indicators.
The threat-level labels were assigned by cybersecurity experts from diverse organizations, reflecting real-world use of the OSINT feeds.
The initial snapshot contained 2005 MISP events and was processed through a multi-stage pipeline before experimentation.

A. Dataset extraction and curation

Dataset curation reduces heterogeneous MISP event structures into compact representations while filtering unusable labels and overly long examples. From 2005 raw events, the process yields 1,622 labeled events for stratified evaluation.

MISP data contains data-layer and context-layer elements, including events, attributes, objects, tags, and Galaxy clusters.
Events vary from a few attributes to several hundred and often contain redundant or low-value technical fields, motivating schema reduction and normalization.
Curation retains event descriptors, tags, Galaxy-cluster names, and selected indicator fields while removing stylistic and sharing-related metadata.
The adapted JSON representation streamlines event structure without discarding events, while preserving contextual information and indicators relevant to severity assessment.
Events with missing or invalid threat levels and examples exceeding the token-length threshold were removed before evaluation.
1,622 labeled events remained after excluding 73 “Undefined” cases and 310 events exceeding the length threshold.

C. Textual Representation

The paper converts MISP JSON events into structured textual representations that preserve their contents while removing formatting overhead. This preprocessing makes inputs substantially more compact and reduces the long tail of extremely large events.

The textual representation preserves event contents while organizing details into headings and bullet points.It includes core event details, tags, indicators of compromise, and objects.
Removing JSON artifacts such as braces and quotation marks considerably decreases token count.
Original MISP files contain a long tail of events exceeding one million tokens.
The textual representation is the most compact format, with most events ranging from a few hundred to a few thousand tokens.Simplification concentrates most events around 5000 tokens, while textual inputs are shorter.

IV. METHODOLOGY

The methodology evaluates LLM threat-level determination with a constrained zero-shot prompt that defines three severity classes and requires machine-readable JSON output. Few-shot prompting is intentionally omitted because long MISP events could consume context and risk truncation.

Zero-shot evaluation uses a prompt that explicitly defines high, medium, and low threat levels.
The prompt requires one valid JSON object containing exactly one label key.
The label value must be exactly high, medium, or low.
Few-shot experiments are omitted because MISP events are often lengthy and demonstrations could reduce available context and risk input truncation.

B. Supervised Fine-Tuning

Supervised fine-tuning uses LoRA with the same task instruction as zero-shot evaluation and optimizes only the assistant’s label-only JSON response. The experiments compare LoRA configurations and full versus class-balanced training data.

LoRA fine-tuning formats each instance as a system-user conversation with a label-only JSON assistant target.
Response-only loss restricts optimization to the assistant output.
The study varies LoRA rank r ∈ {8, 16} and scaling factor α ∈ {32, 64}.
Training compares the full dataset with a downsampled dataset balanced across labels by matching the least common High class.
A preliminary configuration with r = 32 and α = 128 produced degraded performance.

C. Experimental Setup

The experiments evaluate instruction-tuned open-source LLMs using accuracy and macro-F1, with macro-F1 as the main metric because of class imbalance. Zero-shot models generally perform poorly, while balanced-data fine-tuning substantially improves results.

Macro-F1 is the main comparison metric because it gives equal weight to classes in the imbalanced dataset.
Zero-shot models generally struggle to infer threat levels without task-specific adaptation.
0.271 macro-F1 is achieved by the majority Low-class baseline, which only two zero-shot models exceed.Command-R+ reaches 0.313 macro-F1 and Llama-3.1-8B reaches 0.277 with structured text input.
Balanced fine-tuning increases macro-F1 by 20–35 points relative to zero-shot baselines across all models.
Full-dataset fine-tuning degraded performance through overfitting on the dominant Low class.
0.574 macro-F1 and 0.704 accuracy are achieved by Llama-3.3-70B with textual input.Command-R+ follows with 0.554 F1 and 0.698 accuracy, while Llama-3.1-8B reaches 0.520 F1 and 0.675 accuracy.

VI. DISCUSSION

Zero-shot instruction-tuned LLMs struggle with threat-level determination, while fine-tuning improves prediction distributions and performance. Model size helps, but task-specific supervision has the stronger effect and reduces the gap between larger and smaller models.

Zero-Shot Performance: Zero-shot models generally fail to surpass a majority-class baseline and tend to overestimate threat severity.They often predict High or Medium even for low-risk events.
Fine-Tuning Effects: Fine-tuning makes predicted class distributions more closely match the test set, where Low is the majority class.
Model Performance: 0.574 macro-F1 is achieved by Llama-3.3-70B after fine-tuning, followed by Command-R+ at 0.554 and Llama-3.1-8B at 0.520.In zero-shot evaluation, Command-R+ leads with 0.313 macro-F1.
Model Performance: +0.350 F1 is gained by Gemma-3-1B, while Gemma-3-12B gains +0.288 after fine-tuning.The gap between 70B and 8B models shrinks to just over 5 points after task-specific training.
Implications: Fine-tuning on task-specific data is the dominant performance factor, allowing smaller tuned models to approach much larger models at lower computational cost.The authors also identify threat-level determination as subjective and operational, extending beyond pattern recognition.

B. Effect of Input Representation

Textual input representations consistently improve macro-F1 over JSON across zero-shot and fine-tuned settings, although gains can coincide with lower accuracy for the Gemma models. Balanced training data is also necessary because full-dataset fine-tuning overfits to the dominant Low class.

Input Representation: Textual representations improve macro-F1 over JSON in every evaluated setting.Zero-shot gains range from +0.027 for Llama-3.3-70B to +0.184 for Gemma-3-12B.
Input Representation: Text input also improves accuracy, while the Gemma models can show lower accuracy alongside higher macro-F1.This pattern suggests less over-prediction of the majority class and more balanced predictions across threat levels.
Input Representation: Concise textual formatting of MISP events is a central design choice for robustness in LLM-based CTI classification.
Training Data Balance: Over 90% of test cases are assigned the dominant Low class when models are fine-tuned on the imbalanced full dataset.This produces deceptively high accuracy but poor macro-F1.
Training Data Balance: 111 instances per threat level are used in the balanced fine-tuning dataset, producing more robust models and substantial macro-F1 improvements.
LoRA Configuration: (r = 16, α = 64) performs best in the limited LoRA search conducted on Llama-3.1-8B and Llama-3.3-70B.The remaining models use this setting directly because of resource constraints.

D. Comparison with other CTI Tasks

Threat-level determination is evaluated as a more difficult CTI task than those in CTIBench, while the proposed benchmark shows that fine-tuning improves performance but does not yet support reliable operational deployment. Future work targets broader data and knowledge-graph-based representations.

Comparison with Other CTI Tasks: Threat-level determination generally achieves lower performance than the tasks evaluated in CTIBench.In the authors’ comparison, CTIBench accuracy across three tasks exceeds 0.36, while unfine-tuned Llama-3.3 models reach only 0.23–0.30 in accuracy and macro-F1.
Comparison with Other CTI Tasks: After fine-tuning, Llama-3.3-70B stands out as the best performer, whereas CTIBench reports ChatGPT-4 as strongest overall.
Contribution: The paper presents a narrow and deep evaluation of LLMs for threat-level determination in cyber threat intelligence.
Conclusion: 0.40–0.58 macro-F1 is achieved after supervised fine-tuning, but overall performance remains insufficient for reliable operational deployment.The paper identifies data quality, model architecture, and domain-specific tuning as areas requiring further research.
Future Work: The planned dataset expansion beyond OSINT feeds aims to capture more diverse adversarial behaviors and support knowledge-graph-based reasoning.The authors intend to use these resources to improve accuracy and interpretability.