Source-linked AI summary
DisCTI: Who Needs to Know Timely? Automated Sector-Aware Cyber Threat Intelligence Dissemination
Fajar Wijitrisnanto, Alsharif Abuadbba, Yansong Gao, Nan Wu
TL;DR
CTI platforms leave much sector-specific contextualization manual, limiting targeted dissemination amid growing threat-data volumes. DisCTI formulates sector mapping as multilabel classification, builds a dataset of 872 labeled events, and applies BERT; it achieves a macro-averaged F1-score of 0.89 at a Hamming loss of 0.055. The findings support the feasibility of automated, sector-aware dissemination, while data availability, interpretability, and real-time scalability remain open challenges.
Problem
CTI platforms lack automated, granular sector-based tagging at scale, forcing analysts to sift through aggregated data to identify relevant intelligence.
Method
DisCTI formulates sector mapping as multilabel classification, constructs 872 sector-labeled TIP events, and evaluates binary classifiers alongside a BERT-based multilabel model.
Results
0.89 macro-averaged F1-score and 0.055 Hamming loss were achieved by the BERT-based model, which delivered the highest overall performance.
Takeaways & Limitations
The results demonstrate the feasibility of transforming CTI dissemination from a largely manual, reactive process into an automated, sector-aware capability.
Takeaways & Limitations
Data availability, model interpretability, and real-time scalability remain open challenges for operational deployment.
Abstract
from arXiv · showhide
The timely dissemination of cyber threat intelligence (CTI) is critical for organizations to mount swift and effective incident response. When valid CTI is delivered to the right sector at the right time, identical attacks can often be contained or mitigated. However, today's rapidly expanding CTI landscape overwhelms analysts, who must sift through massive and heterogeneous feeds. Existing platforms such as the Malware Information Sharing Platform (MISP) provide sector tagging features (e.g., energy, finance, government), but in practice, these remain largely unmapped (98% of events are left uncategorized). This lack of automated and timely sector mapping severely limits the operational value of shared intelligence, leaving organizations that belong especially to the critical information infrastructure sector exposed. To address this gap, we formulate sector-targeted CTI dissemination as a multilabel classification problem. Leveraging deep field knowledge of CTI structures and sector-specific threat patterns, we construct a novel data set of 872 sector-labelled CTI events from a threat intelligence platform (TIP). We then apply BERT, a transformer-based model, to automate the mapping of CTI events to sectors. Using the structured threat information expression (STIX) format for cross-platform interoperability, our approach achieves a macro-averaged F1-score of 0.89 at a Hamming loss of 0.055 on the custom dataset, i.e. 94.5% of individual sector-label assignments are correct. These results not only demonstrate the feasibility of sector-aware, automated CTI dissemination but also highlight how embedding expert field knowledge into machine learning design fills a crucial gap in the threat intelligence pipeline, enabling faster and context-relevant defensive action.
1. Introduction
CTI dissemination is valuable only when intelligence reaches the relevant stakeholders promptly, yet growing data volumes and missing sector tags make targeted delivery difficult. DisCTI addresses this gap with multilabel classification, a sector-labeled dataset, and comparative evaluation of binary and BERT-based approaches.
- Motivation: CTI systematically collects, analyzes, and disseminates information to support attack prevention and timely incident response.
- Motivation: Rapidly increasing threat data and resource-intensive manual analysis can overwhelm analysts and delay effective, sector-relevant decision-making.
- Problem: Missing sector-specific tagging forces organizations to sift through broad CTI feeds instead of focusing on threats affecting their sector.
- Approach: DisCTI formulates sector-targeted CTI dissemination as a multilabel classification problem because events may concern multiple sectors simultaneously.
- Approach: 872 real-world CTI events were collected and processed into a sector-labeled dataset for systematic evaluation of automated dissemination.
- Evaluation: 0.89 macro-averaged F1-score and 0.055 Hamming loss were achieved by BERT, which outperformed the evaluated baseline approaches.
- Implication: The results support automated sector tagging as a means to advance CTI dissemination and enable faster, more targeted defensive responses in critical infrastructure environments.
2. Preliminary & Related Work
Existing CTI platforms and standards support aggregation, interoperability, and analysis, but automated sector-specific contextualization remains limited. The paper frames this gap as multilabel classification and constructs a sector-labeled dataset for model evaluation.
- CTI Standards: STIX standardizes the representation and exchange of CTI, supporting consistent descriptions and automation across threat intelligence platforms.
- Threat Intelligence Platforms: Threat intelligence platforms aggregate, analyze, visualize, and share data from diverse sources, including logs, feeds, and open-source intelligence.
- Dissemination Gap: Platforms generally lack sophisticated automated sector-specific tagging, leaving contextualization largely manual and contributing to information fatigue and delayed responses.
- Dissemination Gap: Only about 2% of MISPPriv security events are tagged according to organization sector, while manual tagging remains labor-intensive and inconsistent.
- Dissemination Gap: Sector categorization can be ambiguous because taxonomy rationale is not always rigorously defined, allowing the same threat to receive different sector labels.
- Problem Formulation: CTI events may simultaneously concern multiple sectors, making sector-aware dissemination inherently a multilabel classification task.
- Problem Formulation: The formal task maps a textual CTI event x to a binary sector-label vector y, with each positive entry indicating relevance to a sector.
- Problem Formulation: A dataset of real-world TIP entries paired with sector-label vectors provides the foundation for evaluating models and selecting an approach for deployment.
3. Data Collection and Construction
The paper constructs a sector-labeled CTI dataset because existing feeds often lack consistent labels for supervised learning. It collects, parses, structures, and processes MISPPriv data into 872 machine-learning-ready events using STIX 2.1 and custom sector information.
- Existing CTI feeds are typically raw and unstructured, lacking consistent sector-specific labels for supervised model training.
- The study transforms raw feeds into a structured, machine-learning-ready dataset of 872 tagged events.
- MISPPriv was selected to support efficient, reliable, and diverse CTI collection from multiple sources.Its stated capabilities include automated collection, reputable contributors, and varied intelligence such as IoCs and threat-actor information.
- STIX 2.1 represents CTI through STIX Domain Objects describing incident elements such as attack patterns, malware, tools, and vulnerabilities.
- MISP’s custom sector SDO supplies sector context that supports more accurate CTI dissemination.The example links a report and indicator object and adds a finance sector label.
4. DisCTI Design
DisCTI is designed as an automated, sector-specific CTI dissemination framework that treats sector assignment as multilabel classification. Its design covers sector selection, data preparation, and binary-classifier or direct multilabel approaches.
- DisCTI targets CTI dissemination across 11 sectors: 8 Indonesian CII sectors plus industry, education, and media.The eight government-defined sectors are government, defense, transportation, finance, health, energy, ICT, and agriculture.
- Each CTI sample may belong to multiple sectors, so the task is formulated as multilabel classification.
- The framework compares collections of sector-specific binary classifiers with a single model performing multilabel classification directly.
- The design includes preprocessing, tokenization, and formatting inputs as PyTorch tensors for model training and evaluation.
- Parallel binary classifiers treat labels independently, whereas sequential classifiers use earlier predictions as features for later labels.The sequential organization can capture label dependencies; the parallel organization does not.
4.3. BERT based Multi-Label Classifier
The BERT-based component uses a pretrained transformer to automate sector tagging and improve CTI-feed relevance. Its effectiveness is accompanied by substantially higher computational demands than simpler models.
- BERT is a pretrained transformer-based model used to automate CTI sector tagging.The architecture is based on multi-head attention.
- 25 epochs required approximately 125 minutes on a T4 GPU with 15GB of VRAM.The reported average was 5 minutes per epoch.
- BERT-based models use more memory and runtime than simpler models because of their larger parameter count and multi-head attention.The paper says efficiency should be evaluated against available hardware, especially for larger datasets or real-time applications.
4.4. Implementation: Data Preprocessing
The preprocessing pipeline converts CTI records into numerical multilabel inputs by encoding sector labels, combining relevant text fields, cleaning IoC content, and splitting the data for evaluation.
- Sector labels are identified and converted into numerical binary vectors for multilabel model processing.Each vector position indicates whether a sector label is present or absent.
- The report name, publisher, and IoC columns are concatenated into an event-info field, while publication date is discarded.
- IoC cleaning removes noise through whitespace normalization, special-character removal, lowercasing, and lemmatization.The stated goal is higher-quality, more uniform model input with a reduced vocabulary.
- The 872 records are split 75:25 into roughly 654 training events and 218 validation events.
4.5. Training: Parallel Binary Classifiers and Sequential Binary Classifiers
The paper trains multilabel classifiers using TF-IDF representations, evaluating independent binary models and classifier chains that incorporate label dependencies. Gaussian Naive Bayes supports parallel classification, while Random Forest with ClassifierChain supports sequential prediction.
- TF-IDF Vectorization: TF-IDF converts cleaned and lemmatized CTI text into numerical vectors using term frequency and inverse document frequency.The vectorizer is fitted on training text and transforms both training and validation data.
- Parallel Binary Classifiers: Parallel binary classifiers train one independent binary classifier per sector label.This approach treats each label as a separate classification problem after TF-IDF vectorization.
- Parallel Binary Classifiers: Gaussian Naive Bayes models are trained through BinaryRelevance, with each classifier independently predicting one sector label.The TF-IDF vectors serve as input features for the multiple Gaussian Naive Bayes classifiers.
- Sequential Binary Classifiers: Sequential binary classifiers form a chain in which each classifier uses preceding predictions as additional features.This design captures dependencies among sector labels and may improve overall multilabel performance.
- Sequential Binary Classifiers: Random Forest classifiers are linked with ClassifierChain, using TF-IDF vectors and a sequence-determined label order for sequential prediction.The chain is fitted on the training data and produces predictions step by step.
4.6. Training: BERT Model
The BERT training pipeline tokenizes CTI text into padded inputs, generates contextual embeddings, and fine-tunes a multilabel classifier. Training uses AdamW, a linear learning-rate scheduler, BCE loss, and validation across 25 epochs.
- Tokenization and Embedding: BERT tokenization converts CTI text into WordPiece subword units, supporting out-of-vocabulary terms and nuanced meanings.The Hugging Face tokenizer adds [CLS], [SEP], and [PAD] tokens, with sequences limited to 512 tokens.
- Tokenization and Embedding: BERT converts tokenized inputs into dense vectors that capture semantic information through its pretrained transformer architecture.The resulting embeddings are fed into BERT and fine-tuned for CTI dissemination.
- Tokenization and Embedding: Attention masks distinguish actual tokens from padding tokens before input IDs and masks are converted into PyTorch tensors.These tensors provide the required input format for the Transformers implementation.
- Model Configuration: The multilabel BERT model sets its output dimension to the number of dataset classes.It is initialized from a pretrained Transformers model and configured for multilabel classification.
- Training Configuration: AdamW and a linear learning-rate scheduler are used to support transformer training, convergence, and overfitting control.The scheduler gradually adjusts the learning rate during training.
- Training Process: 25 epochs with batch size 6 train the model using binary crossentropy loss, followed by validation after each epoch.Validation excludes backpropagation and supports progress monitoring and hyperparameter adjustment.
4.7. Evaluation Metrics
The evaluation uses multilabel metrics that distinguish individual label errors from precision, recall, and their harmonic mean. The paper specifically interprets 1 − Hamming Loss as label-wise accuracy rather than exact-match accuracy.
- Hamming Loss: Hamming loss measures the fraction of sector-label assignments predicted incorrectly across samples and labels.Its formula averages the indicator of disagreement between predicted and true labels over n samples and L labels.
- Hamming Loss: Label-wise accuracy equals 1 − Hamming Loss and measures the fraction of individual sector-label assignments predicted correctly.This differs from exact-match accuracy, which requires every label for a sample to be correct simultaneously.
- Precision: Precision measures the proportion of true-positive predictions among all positive predictions made by the model.It is calculated from true positives and false positives.
- Recall: Recall measures the proportion of true-positive predictions among all actual positives in the dataset.It is calculated from true positives and false negatives.
- F1-score: F1-score is the harmonic mean of precision and recall, balancing the two measures in one metric.The metric summarizes both types of performance rather than treating precision or recall alone as sufficient.
5. Experiments
The experiments compare parallel binary, sequential binary, and BERT approaches for automated multilabel sector tagging of CTI. BERT provides the strongest overall performance, although its averages exclude the media sector.
- The experiments address automating sector-specific CTI tagging and identifying the best-performing machine learning model.
- BERT handles CTI’s multi-sector nature more effectively than binary classifiers and achieves better precision, recall, and F1 scores.Binary classifiers performed adequately in simplified settings but were limited for multi-sector tagging.
- Parallel Binary Classifiers: 0.9158 accuracy and 0.0842 Hamming Loss were achieved by the Parallel Binary Classifiers model using Gaussian Naive Bayes.Agriculture had the highest F1-score at 0.941, while Defense had the lowest at 0.605.
- Sequential Binary Classifiers: 0.9324 label-wise accuracy and 0.0676 Hamming Loss were achieved by the Sequential Binary Classifiers model using Random Forest.Transport had the highest F1-score at 0.946; Defense had the lowest at 0.769.
- Model comparison: Sequential classification increased macro-averaged precision from 0.745 to 0.941 but reduced macro-averaged recall from 0.856 to 0.769 relative to Parallel classification.Government precision rose from 0.793 to 0.889, while recall fell from 0.823 to 0.709.
- BERT: 0.89 macro-averaged F1-score and 0.055 Hamming Loss were reported for BERT, with 94.5% of individual sector-label assignments correct.The BERT evaluation covers 10 of 11 sectors because media is excluded, so its averages are not strictly comparable with the other models.
6. Discussion
The discussion frames DisCTI as an operational response to manual CTI triage and examines deployment, human–machine collaboration, and dataset foundations. It also identifies efficiency, interoperability, explainability, and scale as practical considerations.
- Operational implications: DisCTI automates sector tagging to reduce manual triage and alert fatigue for agencies, ISACs, and critical infrastructure operators.
- Deployment considerations: Production deployment of BERT introduces computational and latency constraints for large-scale, real-time CTI dissemination.DistilBERT or quantized transformer variants are proposed as lighter alternatives, while STIX/TAXII integration is identified for interoperability.
- Human–machine collaboration: BERT’s attention mechanisms can support explainable visualization, allowing analysts to validate, refine, or override model outputs.The discussion presents this as a basis for combining machine efficiency with human expertise.
- Dataset: The 872-event sector-labeled dataset provides a reproducible foundation for supervised learning and future benchmarking despite its modest initial size.The dataset is described as diverse and realistic, capturing heterogeneous threat data across critical sectors.
7. Conclusion
The study proposes and validates DisCTI for automated, sector-specific CTI dissemination. BERT achieves the highest overall performance, while data availability, interpretability, and real-time scalability remain open challenges.
- DisCTI is a deep-learning framework for automated sector-specific dissemination of cyber threat intelligence.
- 0.89 macro-averaged F1-score and 0.055 Hamming Loss were achieved by the BERT-based model.
- Embedding domain expertise within AI architectures is associated with operational efficiency and strategic impact for national cyber defense.
- Data availability, model interpretability, and real-time scalability remain open challenges for transitioning DisCTI from a research prototype to an operational system.