PaperlayerView exact revision on arXiv

Source-linked AI summary

AnnoCTR: A Dataset for Detecting and Linking Entities, Tactics, and Techniques in Cyber Threat Reports

Lukas Lange, Marc Müller, Ghazaleh Haratinezhad Torbati, Dragan Milchevski, Patrick Grau, Subhash Pujari, Annemarie Friedrich

arXiv:2404.07765v1cs.CLcs.AIcs.CRcs.LG

TL;DR

Cyber threat intelligence depends on extracting information from unstructured reports, while NLP work in this area and clean labeled data remain limited. AnnoCTR contributes openly licensed reports annotated with entities, temporal expressions, and explicit or implicit cybersecurity concepts, and the paper evaluates neural models for these annotations. The reported results include entity-recognition and technique-identification scores, while the authors note constraints on transfer evaluation and computational resources.

  • Problem

    Extracting threat information from unstructured reports is tedious and time-critical, and clean labeled data for reproducible cyber threat intelligence studies is difficult to obtain.

    Source: §1 · p. 1

  • Method

    The authors construct AnnoCTR with entity, temporal, and cybersecurity-concept annotations and evaluate neural models on NLP tasks based on the dataset.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    General-purpose entity recognition reaches up to 70% macro-average F1, while domain-fine-tuned entity disambiguation identifies document-level techniques at around 65% micro-F1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that AnnoCTR lays a foundation for cybersecurity-specific NLP models using freely available and permissively licensed data.

    Source: §6 · pp. 8–9

    Experiments center on AnnoCTR, broader multi-task or transfer learning was limited by licensing, some linking models assume gold-standard entities, and neural training can require up to 12 GPU hours.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Monitoring the threat landscape to be aware of actual or potential attacks is of utmost importance to cybersecurity professionals. Information about cyber threats is typically distributed using natural language reports. Natural language processing can help with managing this large amount of unstructured information, yet to date, the topic has received little attention. With this paper, we present AnnoCTR, a new CC-BY-SA-licensed dataset of cyber threat reports. The reports have been annotated by a domain expert with named entities, temporal expressions, and cybersecurity-specific concepts including implicitly mentioned techniques and tactics. Entities and concepts are linked to Wikipedia and the MITRE ATT&CK knowledge base, the most widely-used taxonomy for classifying types of attacks. Prior datasets linking to MITRE ATT&CK either provide a single label per document or annotate sentences out-of-context; our dataset annotates entire documents in a much finer-grained way. In an experimental study, we model the annotations of our dataset using state-of-the-art neural models. In our few-shot scenario, we find that for identifying the MITRE ATT&CK concepts that are mentioned explicitly or implicitly in a text, concept descriptions from MITRE ATT&CK are an effective source for training data augmentation.

1. Introduction

Cyber threat reports contain information needed for cyber threat intelligence, but extracting it is time-consuming and clean labeled data is difficult to obtain. AnnoCTR provides openly licensed reports with detailed entity and cybersecurity annotations, and the paper evaluates neural models on tasks built from them.

1. Introduction

  • The dataset addresses the challenge of obtaining clean labeled data for replication, validation, and extensions of cyber threat intelligence studies.Cyber threat reports are unstructured, and retrieving and analyzing their threat information is tedious and time-critical.Source: §1 · p. 1
  • AnnoCTR uses MITRE ATT&CK rather than a custom schema and annotates full documents more finely than prior work that labels documents or out-of-context sentences.Many prior datasets are not clearly licensed, making use difficult or potentially copyright-infringing.Source: §1 · pp. 1–2
  • AnnoCTR contains 400 donated cyber threat reports, with 120 annotated by a domain expert for explicit or implicit MITRE ATT&CK tactics and techniques.Annotations also include locations and organizations linked to Wikipedia and normalized temporal expressions.Source: §1 · p. 2
  • Up to 70% macro-average F1 for general-purpose entity recognition and around 65% micro-F1 for identifying document-level techniques with domain-fine-tuned disambiguation models.The technique-identification models use entity disambiguation models fine-tuned on AnnoCTR.Source: §1 · p. 2
  • The paper provides an openly licensed dataset and proposes neural sequence-tagging, classification, and entity-linking baselines for its annotations.The authors present the resource as a basis for NLP and CTI research and application.Source: §1 · p. 2

2. Related Work

Prior cybersecurity NLP datasets and methods cover varied entities, events, and attack labels, but differ in annotation scope and licensing. This section situates AnnoCTR among those resources and describes MITRE ATT&CK and related systems.

2. Related Work

  • MITRE ATT&CK is a hierarchical knowledge base of adversary tactics and techniques compiled from real-world observations, with textual descriptions for its entries.At the time of writing, its Enterprise taxonomy included 14 tactics, 193 top-level techniques, and 401 subtechniques.Source: §2 · p. 2
  • Earlier MITRE ATT&CK resources include document-level rcATT annotations, whose reports come from sources with unclear licensing.The cited work crawled report references and technical descriptions to create rcATT.Source: §2 · p. 2
  • TRAM is an open-source platform for automating mappings from cyber threat reports to MITRE ATT&CK; its dataset labels sentences with multiple techniques.AnnoCTR and its models aim in part to improve TRAM's functionality.Source: §2 · p. 2
  • Existing manually labeled cybersecurity datasets use varied targets, including malware attributes, cybersecurity entities, event types and arguments, named entities, and cyber-attack mentions.The cited examples include MalwareTextDB, Hanks et al.'s CTI blog annotations, CASIE, CySecED, Kim et al., and a dataset of cyber-attack tweets.Source: §2 · p. 2
  • Prior cybersecurity NER methods range from feature-based maximum-entropy and SVM models to recurrent and convolutional neural networks, CRFs, and XLM-R.Related work also explores constructing or enhancing cybersecurity knowledge bases from text.Source: §2 · pp. 2–3

3. AnnoCTR Dataset

AnnoCTR contains 400 vendor-donated cyber threat reports with general-world entity annotations and a cybersecurity-specific annotation layer on 120 reports. The paper describes its annotation scheme, corpus scale, and agreement checks.

3.1. Source of Texts and Preprocessing

  • The 400 reports were published from March 2013 to February 2022 and republished under CC-BY-SA 4.0 with vendor permission.Texts were converted to Markdown-like format; custom sentence segmentation and manual boundary correction addressed links and code snippets.Source: §3.1 · p. 3

3.2. Annotation Scheme

  • The annotation scheme covers general entities and cybersecurity concepts, including explicit and implicit tactics and techniques linked to MITRE ATT&CK.Implicit technique mentions can span a sentence; for example, described behavior may map to an ATT&CK technique even when its name is absent.Source: §3.2 · pp. 3–4

3.3. Corpus Statistics

  • The full corpus contains 20,855 annotated mentions, including 13,244 annotations across the 120 reports with both annotation layers.Technique and tactic labels have long-tailed distributions; AnnoCTR is reported as at least comparable in size to MalwareTextDB v2.Source: §3.3 · p. 4

3.4. Annotation Process and Agreement

  • General-entity agreement varied by type: LOC spans had 100% observed agreement, while relaxed agreement for SECTOR mentions was 70.0% and 35.0% for the two annotators.For ORG exact matches, precision was 78.1% and recall 70.4%; relaxed matching raised these to 89.0% and 80.2%.Source: §3.4 · pp. 4–5
  • Cybersecurity-specific labels were produced by one domain expert, and the limited agreement check was explicitly not a proper agreement study.The second annotator had no extended training; average document-level technique F1 was 54%, and the authors say the study likely underestimates achievable agreement.Source: §3.4 · p. 5

4. Task Definitions and Modeling

The paper defines NLP tasks for AnnoCTR and proposes neural baselines for entity recognition, temporal tagging, entity disambiguation, and sentence-level tactic and technique classification.

4.1. Named Entity Recognition

  • NER detects entity mentions, assigns their types with BIO labels, and predicts labels for each real token's first wordpiece.The compared transformer models are BERT, SciBERT, CodeBERT, and RoBERTa, alongside a reimplementation of a BiLSTM-CRF baseline.Source: §4.1 · p. 5

4.2. Temporal Tagging

  • Temporal tagging compares DATE-limited HeidelTime, adapted by retaining rules that improve development performance, with a multilingual neural tagger and normalization model.The neural approach detects TIMEXes and fills XML-like template slots; the experiments replace its tagger with the cybersecurity-specific NER model.Source: §4.2 · p. 5

4.3. Entity Disambiguation

  • Entity disambiguation restricts candidate knowledge-base entries by entity type and compares BLINK's bi-encoder with GENRE's constrained generation.The BLINK cross-encoder was omitted because it provided no benefit in this setting; subtechnique predictions were mapped to parent techniques.Source: §4.3 · pp. 5–6

4.4. Sentence-based Tactic and Technique Classification

  • Sentence-level classification first detects explicit or implicit tactic and technique mentions, then links predicted mention sentences to ATT&CK concepts.The study compares GENRE, TMM's SciBERT-based binary heads, and TRAM's n-gram ensemble with RoBERTa four-way mention-detection classifiers.Source: §4.4 · p. 6

5. Experiments

The experiments evaluate NLP models on AnnoCTR using a temporal train/dev/test split. Domain-specialized models outperform zero-shot models and baselines for document-level ATT&CK detection, while results vary across tasks and entity types.

5.1. Settings

  • The corpus is split 60/15/25% into train/dev/test with a temporal split that places documents from each vendor in every split, simulating training on older reports and inference on newer ones.NER scores average five runs with different random seeds.Source: §5.1 · p. 6

5.2. NER Results

  • RoBERTa performs best for general and cybersecurity entities, improving by 2–3 points over other BERT models; adding 280 documents consistently improves performance by 2 points.NER uses explicit cybersecurity mentions because preliminary experiments found that models did not work well for implicit mentions.Source: §5.2 · p. 7, Table 5 · p. 7

5.3. Temporal Tagging Results

  • Domain adaptation improves both low-scoring out-of-box temporal taggers: reduced cybersecurity rules raise HeidelTime F1 scores by +12.0/10.5/9.2 pp.; cybersecurity NER replacing neural extraction yields +18.5/11.0 pp. and normalization gains +10.6 pp.For HeidelTime, reduced rules decrease false positives, often due to annotation-scheme differences.Source: §5.3 · p. 7, Table 6 · p. 7

5.4. Entity and Concept Disambiguation

  • For Wikipedia linking, GENRE has higher top-1 precision but lower recall than BLINK for ORG and CON; ORG linking is harder than the other entity types.The authors suggest GENRE may prune good candidates too early and attribute ORG difficulty to several likely candidates.Source: §5.4 · p. 7, Table 7 · p. 7
  • For MITRE ATT&CK linking, zero-shot models perform reasonably well on GROUP, MALWARE, and TOOL, but AnnoCTR fine-tuning is essential for tactics and techniques; retrained GENRE leads for techniques.These linking experiments use gold-standard entity spans and types.Source: §5.4 · pp. 7–8, Table 8 · p. 8

5.5. Text-Classification-based Tactic and Technique Disambiguation Results

  • Domain-specialized models outperform zero-shot models and baselines for document-level ATT&CK detection; description- or TRAM-augmented models work best, with sentence filtering best for techniques but worse than the negative-class baseline for tactics.The weaker tactic filtering result is attributed to the sentence classifier having fewer than one-third as many training examples as for techniques.Source: §5.5 · p. 8, Table 9 · p. 8

6. Conclusion and Outlook

The paper presents AnnoCTR as a large-scale cybersecurity dataset annotated with general-world named entities and cybersecurity concepts. Experiments with neural models demonstrate consistent annotation and support freely available, permissively licensed cybersecurity NLP research.

6. Conclusion and Outlook

  • AnnoCTR is a large-scale cybersecurity dataset annotated with general-world named entities and cybersecurity concepts, including tactics and techniques.The paper also proposes NLP tasks and reports experiments using neural transformer models for NER and linking to Wikipedia and MITRE ATT&CK.Source: §6 · p. 8
  • Experiments with neural transformer models for NER and entity and concept linking demonstrate that the corpus is consistently annotated.Source: §6 · p. 9
  • The authors describe the work as a foundation for cybersecurity-specific NLP models using freely available and permissively licensed data.Source: §6 · p. 9

Ethical Considerations

The ethical considerations note annotator consent and compensation, as well as the dual-use risk associated with NLP work.

Ethical Considerations

Limitations

The experiments focus on AnnoCTR, and licensing issues limit larger-scale multi-task or transfer learning beyond TRAM. Some linking evaluations assume perfect extraction, while neural-model training can require substantial compute.

Limitations

  • The experiments focus on AnnoCTR, and licensing issues prevented larger-scale multi-task or transfer learning with other datasets, except TRAM.Source: Limitations · p. 9
  • The Section 5.4 linking models take gold-standard entities as inputs, assuming a perfect extraction model.Source: Limitations · p. 9
  • Training a neural model required up to 12 GPU hours, a resource level that may not be available to every person or organization.Source: Limitations · p. 9
  • The models use a snapshot of the knowledge base for linking, although the authors say it can be exchanged or updated to include new entities.Source: Limitations · p. 9

7. Bibliographical References

The section describes preprocessing for cyber threat reports and points to corpus statistics on technique and tactic annotations and vendor-grouped data splits.

A. Preprocessing

  • Articles were converted to Markdown-like text, sentence-segmented with a custom regular-expression tokenizer, and manually boundary-corrected before annotation.The preprocessing addressed links and code snippets that impaired off-the-shelf sentence segmenters.Source: A. Preprocessing · p. 12

B. Corpus Statistics

C. NER results per class

Per-class NER results are reported in Table 11.

C. NER results per class

D. Computational experiments

The computational-experiment material specifies model resources and training costs and identifies the accompanying data-split, annotation-distribution, and sequence-labeling result tables and figures.

D. Computational experiments

  • All neural-model experiments use V100 GPUs.Source: D. Computational experiments · p. 12
  • Base-sized transformer models have 110M parameters and take 20 minutes to train for NER; BLINK encoders have 340M parameters each and take 10 hours.GENRE has 406M parameters and trains for 12 hours; the BiLSTM-CRF NER baseline has 41M parameters and takes 2 hours, while TMM has 145M parameters and takes 1 hour.Source: D. Computational experiments · p. 12
  • Table 10 presents detailed information on the corpus data splits.Source: Table 10 · p. 13
  • Figures 3 and 4 show technique-link and tactic-link distributions; Figure 3 covers techniques occurring at least five times and notes 136 less-frequent instances.Source: Figure 3 · p. 13, Figure 4 · p. 14
  • Table 11 reports sequence-labeling average micro F1 and standard deviation over five runs, plus recall and precision for the best model, RoBERTa.+X marks models trained with extra data.Source: Table 11 · p. 14

Related papers

Loading 2404.07765v1…