PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Based Weak Annotation for Cybersecurity Tweet Classification: A Teacher-Student Framework

Ferry Afit Kurniawan, Kazushi Okamoto, Koki Karube, Kei Harada, Atsushi Shibata

arXiv:2610.21254v1cs.CL

TL;DR

Expert annotation cannot keep pace with cybersecurity tweet volume, while the reliability and downstream value of LLM-generated labels for cybersecurity-specific PLMs remain underexplored. The paper evaluates LLM Teachers and uses a selected model to label tweets for Student training. LLM-only models recover approximately 98% of human-only F1 on average across four PLMs.

  • Problem

    The reliability of LLM-generated weak labels for cybersecurity tweet classification and their usefulness for training cybersecurity-specific PLMs remain insufficiently explored.

    Source: §I · p. 1

  • Method

    The framework evaluates candidate LLM Teachers, uses the selected model to weakly label unlabeled cybersecurity tweets, and fine-tunes cybersecurity-specific PLMs.

    Source: §I · p. 1

  • Results

    Approximately 98% of human-only F1 was recovered on average across four cybersecurity-specific PLMs trained with LLM-only labels.

    Source: §I · p. 2

  • Takeaways & Limitations

    LLM-based weak annotation offers a practical approach to reducing manual annotation effort while maintaining competitive downstream performance.

    Source: §V · p. 6

    The study covers binary relevance classification and English-language tweets, and its corpus from 112 curated X accounts may miss discussions from newly emerging sources.

    Source: E. Limitations · pp. 5–6

Abstract

from arXiv · show

Cyber threat intelligence (CTI) relies on the timely identification of emerging threats, enabling organizations and security analysts to support situational awareness and decision-making. Social media platforms such as X (formerly Twitter) have become valuable CTI sources because vulnerabilities and attack activities are often discussed there before appearing in formal reports. However, the rapid growth of cybersecurity discussions on social media has created more content than domain experts can realistically review and annotate, making labeled data a major bottleneck for developing supervised learning approaches. While large language models (LLMs) have shown promise as weak annotators in other domains, their effectiveness for cybersecurity tweet classification and downstream training of domain-specific pre-trained language models (PLMs) remains underexplored. To address this challenge, we propose a Teacher-Student weak supervision framework in which a selected LLM serves as a Teacher to generate weak labels for unlabeled cybersecurity tweets, enabling the training of cybersecurity-specific PLMs without large-scale manual annotation. Experimental results show that PLMs trained solely on LLM-generated labels consistently achieve F1 scores above 0.92, remaining within 3 percentage points of fully human-supervised models across all evaluated architectures. These findings suggest that LLM-based weak annotation offers a practical alternative for constructing cybersecurity-specific training data when expert annotation resources are limited.

I. INTRODUCTION

Cybersecurity tweets offer timely CTI, but their volume makes expert annotation a bottleneck, while LLM weak labels’ reliability and value for training cybersecurity-specific PLMs remain underexplored. The paper evaluates LLM Teachers and trains Student PLMs under alternative supervision settings.

I. INTRODUCTION

  • Cybersecurity tweets can surface vulnerabilities and attack activity before formal advisories, supporting timely CTI and situational awareness.Researchers, vendors, and practitioners share vulnerabilities, malware campaigns, exploit activity, and indicators of compromise on social platforms.Source: §I · p. 1
  • Tweet volume exceeds experts’ capacity for manual annotation, creating a labeled-data bottleneck for supervised cybersecurity classification.The tweets are also noisy and brief.Source: §I · p. 1
  • Whether LLM-generated labels reliably classify cybersecurity tweets and train cybersecurity-specific PLMs remains insufficiently explored.LLMs have generated weak annotations for text classification tasks in other domains.Source: §I · p. 1
  • The framework evaluates candidate LLM Teachers, selects one to label unlabeled tweets, then fine-tunes cybersecurity-specific PLMs under human-only, LLM-only, and hybrid supervision.Its three research questions cover Teacher reliability, weak-label utility for Student training, and downstream comparisons across supervision settings.Source: §I · p. 1
  • The study evaluates three LLM annotators across public benchmarks and a newly collected gold-standard dataset, and proposes a scalable pipeline to reduce extensive manual annotation.The pipeline uses LLM-generated weak labels to construct training data for cybersecurity-specific PLMs.Source: §I · p. 2
  • Across four cybersecurity-specific PLMs, LLM-only training recovers approximately 98% of human-only F1 on average.The authors describe this as competitive downstream performance when annotation resources are limited.Source: §I · p. 2

II. RELATED WORK

Prior work developed cybersecurity tweet classifiers and LLM-based weak annotation methods, but the two lines had not been systematically brought together to train cybersecurity-specific PLMs.

A. Cybersecurity Tweet Classification

  • Cybersecurity tweet classification research progressed from deep neural models and novelty detection to transformer fine-tuning, self-training, and prompt-based learning.These studies mainly emphasize model design or task-specific supervision rather than LLM-generated weak labels for cybersecurity-specific PLMs.Source: A. Cybersecurity Tweet Classification · p. 2

B. LLM-Based Weak Supervision

  • General-domain NLP studies explored LLM annotation through direct labeling, explanation-guided prompting, distillation, and lower-cost annotation.Reported work includes ChatGPT annotation quality comparable to or better than crowd workers across several classification tasks.Source: B. LLM-Based Weak Supervision · p. 2
  • Hybrid annotation approaches combine human–LLM collaboration or selective human review with LLM-generated labels.MEGAnno+ offers an interactive annotation environment, while another approach uses verification mechanisms.Source: B. LLM-Based Weak Supervision · p. 2

C. Research Gap and Positioning

The paper addresses the untested intersection of LLM-generated weak labels and cybersecurity-specific PLMs by evaluating Teachers and comparing human-only, LLM-only, and hybrid supervision.

C. Research Gap and Positioning

  • Cybersecurity tweet studies had focused on architectures or self-training and prompting, while LLM annotation studies largely addressed general-domain NLP.The reliability of LLM labels for training cybersecurity-specific PLMs and the value of combining human and LLM labels had not been systematically investigated.Source: C. Research Gap and Positioning · p. 2

A. Overall Framework

  • The framework prepares public and newly collected data, selects a Teacher, labels unlabeled tweets, fine-tunes cybersecurity-specific PLM Students, and compares supervision settings.The settings are human-only, LLM-only, and hybrid.Source: A. Overall Framework · p. 2

B. Datasets

The new corpus was collected from cybersecurity-related X accounts, manually annotated for a gold-standard benchmark, and divided into held-out, Teacher-selection, and unlabeled subsets.

1) Data Collection:

  • From 17,581 tweets collected from 112 cybersecurity-related X accounts, filtering and normalization yielded 13,451 unique English-language tweets.The sources included researchers, security vendors, cybersecurity organizations and agencies, and news outlets.Source: 1) Data Collection · p. 2
  • Two cybersecurity-expert annotators independently labeled a stratified sample; agreement was κ = 0.89, and excluding disagreements yielded 1,897 gold-standard tweets.The 103 tweets with differing labels were excluded.Source: 1) Data Collection · pp. 2–3

2) Data Splitting:

  • Of the 1,897 labeled tweets, 400 formed an independent held-out test set and 1,497 remained for human-only and hybrid supervision.From the 1,497-tweet pool, 400 were sampled for Teacher selection; the 11,451 unlabeled tweets were automatically labeled for Student training.Source: 2) Data Splitting · p. 3

3) Public Benchmark Datasets:

The public datasets use different label schemas, so the study harmonizes them into a shared Relevant/Irrelevant classification task. The mapping preserves concrete cyber-threat information as Relevant and groups other content as Irrelevant.

4) Label Harmonization:

  • All three datasets are evaluated under a unified binary task: Relevant tweets contain concrete cyber-threat information, while tweets without it are Irrelevant.CybAttT maps Normal News and High Risk News to Relevant and Not News to Irrelevant; Behzadan maps Threat to Relevant and Business and Irrelevant to Irrelevant.Source: 4) Label Harmonization · p. 3

3) Public Benchmark Datasets:

  • Teacher models were evaluated on the newly collected dataset and the Behzadan and CybAttT public datasets to reduce reliance on any single benchmark.Source: 4) Label Harmonization · p. 3

4) Label Harmonization:

C. Teacher Model Selection

The study evaluates candidate LLM Teachers with a shared zero-shot annotation prompt, uses the selected Teacher to label unlabeled tweets, and trains cybersecurity-specific PLMs under different supervision settings.

C. Teacher Model Selection

  • Three candidate LLMs were evaluated on 1,000 manually labeled samples using the same zero-shot task of classifying tweets as Relevant or Irrelevant.The sample included 150 Relevant and 150 Irrelevant tweets from each public dataset and 400 tweets from the gold-data pool.Source: C. Teacher Model Selection · pp. 3–4
  • The universal zero-shot annotation prompt was used identically across all Teacher candidates and all three datasets.Source: Figure 2 · p. 4

D. Weak Annotation

  • The selected Teacher labeled 11,451 unlabeled tweets, producing 7,275 Relevant and 4,176 Irrelevant examples without manual correction.The weakly labeled corpus was used for LLM-only training and as additional supervision in the hybrid setting.Source: D. Weak Annotation · p. 4

E. Student PLM Training

  • Four cybersecurity-specific PLMs served as Students; human-only, LLM-only, and hybrid training used manual labels, Teacher labels, or both.The models were CTI-BERT, CySecBERT, SecureBERT-2.0, and SecRoBERTa.Source: E. Student PLM Training · p. 4

F. Evaluation Procedure

Teacher annotations are assessed against human labels, while Student PLMs are evaluated on a held-out test set. The procedure compares supervision regimes across paired seed-level F1 scores.

F. Evaluation Procedure

  • Teacher performance on the 1,000-tweet evaluation set was measured using accuracy, precision, recall, and F1, with the highest overall F1 determining the selected Teacher.Source: F. Evaluation Procedure · p. 4
  • Student PLMs were evaluated on a 400-tweet held-out test set that was excluded from training and model selection.Source: F. Evaluation Procedure · p. 4
  • Supervision regimes were compared using an exact permutation Friedman test on five paired seed-level F1 scores, with Holm-corrected Wilcoxon comparisons reported descriptively.Five paired seeds limit attainable exact p-values for the pairwise tests.Source: F. Evaluation Procedure · p. 4

IV. EXPERIMENT AND RESULTS

Across four Student PLMs, LLM-only training approached human-only F1, while Teacher selection favored Llama-3.3-70B on the full evaluation set. Hybrid training exceeded LLM-only mean F1 for every PLM, and pairwise supervision differences were interpreted descriptively.

A. Teacher Model Selection

  • On CybAttT, all three candidate Teachers exceeded 0.82 F1, while all three had lower F1 on Behzadan, mainly associated with lower recall.The authors suggest some harmonized Behzadan Relevant tweets may not meet the prompt’s concrete-threat criterion.Source: A. Teacher Model Selection · p. 4
  • Across 1,000 evaluation tweets, Llama-3.3-70B achieved the highest F1 (0.8436), followed by GPT-4.1-nano (0.8312) and GPT-4o-mini (0.7758).Llama-3.3-70B and GPT-4.1-nano significantly outperformed GPT-4o-mini; their difference was not significant.Source: A. Teacher Model Selection · p. 4

B. Student PLM Classification

  • LLM-only training recovered approximately 98% of human-only F1 across the four PLMs, with a range of 97.6%–98.8%.The results are reported as means across five runs on the held-out test set.Source: B. Student PLM Classification · p. 4, Table IV · p. 5
  • Hybrid training achieved higher mean F1 than LLM-only training for all PLMs and the highest mean F1 for SecRoBERTa (0.9392).The largest mean F1 gap between human-only and LLM-only training was for CTI-BERT: 0.9543 versus 0.9312.Source: B. Student PLM Classification · p. 5, Table IV · p. 5
  • Omnibus tests found differences among supervision regimes for every PLM, but five paired seeds prevented Holm-adjusted pairwise tests from reaching the 0.05 threshold.Pairwise differences were therefore interpreted descriptively based on mean F1.Source: B. Student PLM Classification · p. 5, Table IV · p. 5, Table V · p. 5

C. Size-matched LLM-only Ablation

Human-only labels outperform size-matched LLM-only labels, while the full weakly labeled corpus narrows the gap. The discussion links this improvement to scale and favors Llama-3.3-70B as Teacher for its recall.

C. Size-matched LLM-only Ablation

  • Human-only training exceeded size-matched LLM-only training by 2.9–3.3 percentage points in mean F1, while the full 11,451-tweet corpus narrowed the gap to 1.1–2.3 points.The size-matched subset contained 1,497 tweets; the full corpus was 7.6× the human-only pool.Source: C. Size-matched LLM-only Ablation · p. 5
  • The annotation of the 11,451 unlabeled tweets incurred total API charges of less than US$0.70, as stated for that annotation activity in the cited passage.Source: C. Size-matched LLM-only Ablation · p. 5

D. Discussion

  • Llama-3.3-70B was selected as Teacher for the highest overall F1 and recall, although its F1 did not significantly differ from GPT-4.1-nano (p = 0.92).Higher recall is desirable because the relevance classifier is an initial CTI filter, where missing a genuine threat is considered more costly than flagging additional false positives for review.Source: 1) RQ1: Which candidate LLM provides… · p. 5
  • At matched training-set size, human-only training yielded higher mean F1 across all PLMs, but the gap narrowed with the full weakly labeled corpus.The discussion attributes the improvement to training-set scale rather than per-label accuracy.Source: 2) RQ2: Can LLM-generated weak labels… · p. 5
  • Human-only supervision had the highest mean F1 for most PLMs, while hybrid supervision exceeded LLM-only results for all PLMs but rarely surpassed human-only training.The discussion suggests additional weak labels may introduce unsupported label assignments rather than fabricated content.Source: 3) RQ3: How do LLM-only, human-only,… · p. 5

E. Limitations

The study is limited to binary relevance classification, English tweets, and a dataset drawn from 112 curated cybersecurity-related X accounts. The conclusion presents weak annotation as a practical approach and proposes extending the framework to topic modeling and event detection.

E. Limitations

  • The framework addresses binary Relevant-versus-Irrelevant classification, chosen as the first CTI filtering stage before analyses such as topic modeling or event detection.Source: E. Limitations · p. 5
  • The study includes English-language tweets and a dataset from 112 curated cybersecurity-related X accounts, which may miss discussions from newly emerging sources.The authors suggest broader collection strategies while maintaining data quality.Source: E. Limitations · p. 6

V. CONCLUSION

  • The proposed Teacher-Student framework uses an LLM to generate weak labels for training domain-specific PLMs.Source: §V · p. 6
  • The authors conclude that LLM-based weak annotation offers a practical way to reduce manual annotation effort while maintaining competitive downstream performance.Future work should integrate topic modeling and event detection to construct structured CTI events from weakly labeled social media streams.Source: §V · p. 6
Loading 2610.21254v1…