Source-linked AI summary
SecureBERT: A Domain-Specific Language Model for Cybersecurity
Ehsan Aghaei, Xi Niu, Waseem Shadid, Ehab Al-Shaer
TL;DR
Cybersecurity text contains specialized and ambiguous language that general models may process inadequately, motivating a domain-specific model. The paper develops SecureBERT using cybersecurity pretraining data, a customized tokenizer, and altered pretrained weights, then evaluates it across language tasks. The reported evaluations show stronger cybersecurity-language performance, while the authors note a data-size-related overfitting limitation.
Problem
Cybersecurity text includes uncommon and ambiguous terms, while analysts must transform threat-related language into usable information for security decisions.
Method
SecureBERT combines a BERT-based architecture with 1.1 billion cybersecurity words, customized tokenization, and noise-based adjustment of pretrained weights.
Results
SecureBERT outperforms existing language models on cybersecurity masked-token prediction, and its tokenizer achieves a higher MLM hit rate than the original RoBERTa tokenizer.
Takeaways & Limitations
The evaluations support SecureBERT as a model for capturing cybersecurity language and processing cybersecurity-related NLP tasks.
Takeaways & Limitations
The RoBERTa training data was 25 times larger than SecureBERT's, creating a small-data setting that may cause overfitting.
Abstract
from arXiv · showhide
Natural Language Processing (NLP) has recently gained wide attention in cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber automation. Increased connection and automation have revolutionized the world's economic and cultural infrastructures, while they have introduced risks in terms of cyber attacks. CTI is information that helps cybersecurity analysts make intelligent security decisions, that is often delivered in the form of natural language text, which must be transformed to machine readable format through an automated procedure before it can be used for automated security measures. This paper proposes SecureBERT, a cybersecurity language model capable of capturing text connotations in cybersecurity text (e.g., CTI) and therefore successful in automation for many critical cybersecurity tasks that would otherwise rely on human expertise and time-consuming manual efforts. SecureBERT has been trained using a large corpus of cybersecurity text.To make SecureBERT effective not just in retaining general English understanding, but also when applied to text with cybersecurity implications, we developed a customized tokenizer as well as a method to alter pre-trained weights. The SecureBERT is evaluated using the standard Masked Language Model (MLM) test as well as two additional standard NLP tasks. Our evaluation studies show that SecureBERT\footnote{\url{https://github.com/ehsanaghaei/SecureBERT}} outperforms existing similar models, confirming its capability for solving crucial NLP tasks in cybersecurity.
1 Introduction
Cybersecurity generates large volumes of sensitive text that analysts must interpret for risk management, while general-language models may lack cybersecurity vocabulary and semantic coverage. SecureBERT addresses this gap with domain-specific pretraining, tokenization, and weight adjustment.
- Cybersecurity analysts analyze text to identify vulnerabilities and other threat-related information for monitoring, prevention, and risk control.
- General English models may struggle with cybersecurity terms that are uncommon or ambiguous across domains, restricting comprehension of cybersecurity implications.
- SecureBERT is introduced as a BERT-based language model designed to process texts with cybersecurity implications across tasks including phishing, malware, code, and intrusion analysis.
- The model uses 1.1 billion cybersecurity words, a customized tokenizer, and random noise applied to pretrained weights while preserving general English vocabulary.
- SecureBERT is evaluated on masked language modeling, sentiment analysis, and named entity recognition to assess cybersecurity and general-English processing.
2 Overview of BERT Language Model
BERT pretrains a bidirectional Transformer language model and then adapts it to labeled downstream tasks. RoBERTa modifies this framework, while BERT established strong results across multiple NLP benchmarks.
- BERT uses surrounding words through bidirectional training, allowing contextual representations based on the full sentence rather than only adjacent words.
- BERT uses a Transformer encoder with attention to learn contextual relations between words and subwords while reading the sequence simultaneously.
- Pretraining uses Masked Language Modeling and Next Sentence Prediction on unlabeled data before downstream fine-tuning with labeled data.
- BERT's pretrained parameters are fine-tuned across downstream tasks within a largely unified architecture.
- RoBERTa is described as a robustly optimized BERT derivative that modifies tokenization and architecture and omits the NSP task during training.
3 Data Collection
The authors assembled a large, heterogeneous corpus of cybersecurity text from online resources for SecureBERT pretraining. It combines introductory material with advanced CTI, research, and threat-report content.
- 98,411 online cybersecurity-related texts were collected using a web crawler.
- The corpus contains 1.1 billion words divided into 2.2 million documents averaging 512 words, processed with spaCy.
- Sources include books, blogs, news, security reports, videos, journals, conference papers, white papers, tutorials, and surveys.
- The collection spans basic cybersecurity information, news, Wikipedia, and tutorials alongside CTI, research articles, and threat reports.
4 Methodology
SecureBERT’s methodology combines a cybersecurity corpus, a customized tokenizer, and adjusted pre-trained weights to retain general English while capturing domain-specific meanings.
- Customized tokenizer: The customized tokenizer uses BPE to preserve RoBERTa vocabulary while adding cybersecurity-specific words and subwords.BPE’s character-based encoding can represent input without introducing unknown tokens.
- Customized tokenizer: SecureBERT shares 32,592 tokens with RoBERTa and adds 17,673 tokens unique to the cybersecurity corpus.Examples of unique terms include firewall, ransomware, malware, phishing, and vulnerability.
- Customized tokenizer: Mutual tokens retain RoBERTa’s original indices, while new cybersecurity tokens receive nonconflicting random indices and corresponding weights.This preserves access to pre-trained weights for shared vocabulary while incorporating domain-specific terms.
- Weight adjustments: To reduce overfitting and shift shared-token meanings toward cybersecurity, SecureBERT adds small noise to RoBERTa-derived weights before retraining.The paper describes noise as a regularizer for limited training data and applies random Gaussian noise to embedding vectors.
- Model architecture: SecureBERT uses the RoBERTa-base architecture with twelve hidden transformer layers, twelve attention heads, 768-dimensional hidden states, and 512-dimensional input embeddings.The model is initialized from RoBERTa and trained on the collected cybersecurity corpus with the customized tokenizer.
5 Evaluation
SecureBERT is evaluated on cybersecurity masked-language prediction, homograph handling, tokenizer impact, sentiment analysis, and cybersecurity NER. Across these evaluations, the study reports stronger cybersecurity-context prediction and competitive or superior downstream performance.
- Evaluation setup: SecureBERT is trained with dynamic masking for 250,000 steps and evaluated on cybersecurity MLM, sentiment analysis, and NER.Training used RoBERTa hyperparameters on eight Tesla V100 GPUs; the evaluation targets both cybersecurity text processing and general-language effectiveness.
- Masked Language Model: The cybersecurity MLM test contains 17,341 MITRE-derived records, masking nouns and verbs to measure topN prediction hit rates.The dataset includes 12,721 masked-noun records and 4,620 masked-verb records, with nouns and verbs selected for their semantic importance in cybersecurity contexts.
- Masked Language Model: SecureBERT consistently outperforms RoBERTa-base, RoBERTa-large, and SciBERT on cybersecurity masked-token prediction.The reported comparison covers both masked nouns and masked verbs; the models perform marginally better on verbs than nouns.
- Masked Language Model: SecureBERT correctly interprets advanced cybersecurity contexts involving reconnaissance, hijacking, and DDoS, whereas RoBERTa-large produces different predictions.The paper presents these examples as evidence that model size alone does not ensure desired output for advanced cybersecurity language.
- Ablation Study: For homographs, weight adjustment helps SecureBERT produce cybersecurity-relevant predictions such as DoS, crash, and reboot.Without adjustment, SecureBERT returns words such as problem, disaster, and crashes, while RoBERTa models return medically associated terms for the virus example.
- Ablation Study: The customized tokenizer achieves a higher masked-word hit rate than the original RoBERTa tokenizer by indexing more complete cybersecurity-related tokens.This comparison is presented as evidence for the significance of domain-specific tokenization in domain-specific language modeling.
- Sentiment Analysis: On general English sentiment analysis, SecureBERT performs similarly to SciBERT, while its modified version improves over the raw version by 0.34% accuracy and 0.71% F1-score.The comparison uses fine-tuning on the Rotten Tomatoes dataset and includes RoBERTa-base and SciBERT baselines.
- Named Entity Recognition: In cybersecurity NER, modified SecureBERT outperforms the other evaluated models on MalwareTextDB despite the dataset containing many general-English sentences.The paper frames cybersecurity NER as challenging because publicly available domain-specific data and consensus on entity classes are limited.
6 Related Works
Prior work applied BERT-based language models to scientific, biomedical, clinical, and limited cybersecurity NLP tasks. These studies established domain adaptation and task-specific fine-tuning as relevant approaches, while cybersecurity language modeling remained comparatively limited.
- Scientific and biomedical models: SciBERT applies BERT’s architecture with unsupervised pretraining on a 1.14M multi-domain scientific-literature corpus.Its corpus comprises 18% computer science and 82% biomedical literature.
- Scientific and biomedical models: BioBERT introduced a BERT-based biomedical model and a benchmark spanning named entity recognition, relation extraction, document classification, and question answering.The model focuses particularly on biomedical text and uses publicly available biomedical datasets.
- Scientific and biomedical models: ClinicalBERT adapted BERT to clinical text from the MIMIC-III database.
- Cybersecurity applications: Cybersecurity applications of BERT were limited, including CyBERT for identifying cybersecurity claims in ICS device documents and fine-tuned BERT for hierarchical vulnerability classification.
7 Conclusions and Future Works
The study introduces SecureBERT, a RoBERTa-based transformer language model trained on cybersecurity text using customized tokenization and altered pretrained weights. Evaluation with masked language modeling and named entity recognition produced promising results for understanding cybersecurity language.
- Conclusions and Future Works: SecureBERT is a RoBERTa-based transformer language model trained on 1.1 billion words from online cybersecurity resources.
- Conclusions and Future Works: The model combines customized tokenization with altered pretrained weights to capture contextual relationships and semantic meanings in cybersecurity text.
- Conclusions and Future Works: SecureBERT was evaluated with masked language modeling and named entity recognition, producing promising results in grasping cybersecurity language.