PaperlayerView exact revision on arXiv

Source-linked AI summary

Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion

Cédric Bonhomme, Alexandre Dulaunoy

arXiv:2607.25572v1cs.CR

TL;DR

Automating CVE-to-ATT&CK mapping is difficult because vulnerabilities describe software flaws while techniques describe adversary behavior. This paper trains and rigorously evaluates a supervised classifier, finding that expert-curated labels outperform zero-shot similarity while LLM-generated expansion provides no reliable improvement.

  • Problem

    CVE-to-ATT&CK mapping is difficult to automate at scale because vulnerabilities imply multiple, long-tailed, partly subjective attacker techniques.

  • Method

    The paper trains a supervised multi-label transformer classifier on a curated expert-labeled gold set while keeping automatically derived labels separate.

  • Results

    A gold-trained classifier roughly doubles recall@5 over zero-shot similarity, while LLM labels agreeing with experts at approximately 0.39 provide no reliable gain across 100–984 added CVEs.

  • Takeaways & Limitations

    Expert-curated data improves mapping performance, whereas LLM-labeled data does not, and checkpoint selection on small test splits can produce misleading conclusions.

  • Takeaways & Limitations

    The small, nonrepresentative gold set over-represents exploited vulnerabilities, so results may not transfer unchanged to never-exploited CVEs or the full ATT&CK matrix.

Abstract

from arXiv · show

We present a reproducible pipeline for mapping Common Vulnerabilities and Exposures (CVEs) to MITRE ATT&CK Enterprise techniques from free-text vulnerability descriptions. Rather than relying on the CWE->CAPEC->ATT&CK derivation chain, whose table-expansion artifacts we quantify, we train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings. The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline and improves every ranking metric. We then investigate whether LLM-assisted labeling can extend the gold dataset. Initial experiments suggest contradictory conclusions: a single run indicates degraded performance, while averaging over five random seeds suggests a small gain. However, an independent replication and an expansion-size study (100--984 additional CVEs) show that the apparent improvement is an evaluation artifact. LLM-generated labels, with approximately 0.39 agreement with expert annotations, provide no reliable improvement at any expansion size and reduce rare-technique coverage at around 1,000 added CVEs (macro-F1 decreases by 0.04). The root cause is evaluation noise. Selecting checkpoints on a small test split effectively maximizes over many noisy evaluations, producing recall@5 differences of up to 0.05 between otherwise identical runs. Using a corrected protocol based on validation-split checkpoint selection, the gold-only model achieves recall@5 of (0.673 \pm 0.019), and repeating the decisive experiment confirms the null result for LLM expansion. A final scaling study shows that additional expert-curated data consistently improves performance, whereas LLM-labeled data does not, indicating that the classifier is limited by label quality rather than dataset size. All datasets, models, code, and training logs are publicly released.

1 Introduction

The paper frames CVE-to-ATT&CK mapping as a difficult, multi-label task connecting software flaws to adversary behavior, and proposes a supervised approach grounded in expert-curated labels. It contributes a provenance-tiered dataset, stronger classification, quantified derivation-chain noise, and a reproducible analysis of LLM-assisted label expansion.

  • Motivation: CVE-to-ATT&CK mapping helps defenders connect vulnerability feeds to the adversary techniques enabled by flaws and to detection coverage.
  • Challenges: The task is difficult because CVEs can imply multiple techniques, technique frequencies are long-tailed, and expert analysts may disagree about subjective mappings.
  • Challenges: The CWE→CAPEC→ATT&CK shortcut is unsuitable as a training target because its labels are dominated by quantified table-expansion artifacts.
  • Approach: The study trains a multi-label classifier on expert CTID mappings and benchmarks it against a training-free semantic-similarity baseline.The dataset keeps expert gold labels separate from weak automatically derived labels.
  • Contributions: The supervised classifier roughly doubles recall@5 versus a zero-shot similarity baseline and improves every ranking metric under an analyst-oriented evaluation protocol.
  • Contributions: ≈0.39-agreement LLM labels provide no reliable gain across 100–984 added CVEs and degrade rare-technique coverage at the largest expansion size.The paper attributes conflicting expansion verdicts to best-checkpoint selection on a small test split amplifying run-to-run noise, and releases code, datasets, and models for reproduction.

2 Background and Related Work

This work maps CVEs to ATT&CK techniques using version-normalized identifiers and expert CTID labels that separate exploitation, primary impact, and secondary impact. It contrasts derived and text-based prior approaches while testing whether LLM-generated labels improve supervised learning despite possible noise.

  • ATT&CK representation: ATT&CK organizes adversary behavior into tactics, techniques, and sub-techniques, requiring identifier normalization across evolving versions.The pipeline uses revoked-by relationships in ATT&CK STIX data to normalize identifiers to a single reference version.
  • Label methodology: CTID mappings decompose each CVE-to-ATT&CK assignment into exploitation, primary-impact, and secondary-impact slots.Gold labels come from two CTID artifacts: the 2021 attack-to-CVE mapping and Known-Exploited-Vulnerabilities mappings.
  • Related work: Prior automated methods either derive techniques through CWE→CAPEC→ATT&CK tables or learn mappings from vulnerability text.Derived approaches inherit noise from underlying mapping tables, while text-based approaches include semantic-similarity systems and supervised BERT fine-tuning.
  • Study motivation: The study differs by using expert CTID labels, multi-label ranking for analyst review, and an explicit evaluation of LLM-assisted label expansion.The expansion experiment tests whether cheaply generated labels are accurate enough to improve a supervised model or whether label noise outweighs added coverage.

3 Dataset Construction

The dataset uses manually curated MITRE CTID mappings as gold training targets, while keeping CWE→CAPEC→ATT&CK labels separate as a weak signal because their broad fan-out can encode mapping-table artifacts rather than vulnerability behavior.

  • Gold-set construction: ~1,260 CVEs are assembled from two CTID sources: the 2021 attack to cve mapping and the Mappings Explorer KEV set.The sources follow a methodology assigning up to three slots: exploitation technique, primary impact, and secondary impact.
  • Gold-set construction: Human ground truth comes from MITRE analyst mappings; the pipeline only fetches, normalises, merges, and splits these existing annotations.No CVEs are annotated by the authors.
  • Gold-set construction: Cross-release identifier normalisation prevents one behavioral concept from being split across revoked and successor technique labels.Without normalisation, fragmented labels would inflate the vocabulary and create evaluation mismatches.
  • Derived-label evaluation: 88.3% of CVEs receive at least one derived technique, but the median labeled CVE receives 8 and the mean receives 13.3.The CVE2CAPEC CVE-2024 database contains 39,156 CVEs, with 7,381 receiving twenty or more techniques.
  • Derived-label evaluation: Derived labels are excluded from training because examples such as CVE-2024-21732 receive techniques unrelated to their described cross-site-scripting behavior.The derived labels remain in a separate column for filtering, inference-time priors, and comparison with analyst judgment.

4 Supervised Classifier

The supervised classifier treats CVE-to-ATT&CK assignment as multi-label prediction, using an independently scored transformer encoder trained with weighted binary cross-entropy. It also addresses label imbalance and operational deployability, while checkpoint-selection split choice is a critical methodological concern.

  • Model formulation: The model uses a transformer encoder with a sigmoid head and binary cross-entropy, making each ATT&CK technique an independent 0/1 decision.This reflects that one vulnerability can imply several techniques across exploitation and impact.
  • Operational motivation: A single forward pass assigns all technique scores, supporting continuous operation by national CSIRTs, small security teams, and open-data publishers.The stated motivation is deployability rather than a measured energy-saving ratio.
  • Label-distribution handling: Per-label positive weights counter long-tailed imbalance, while collapsing sub-techniques to parent techniques regularises the label space.Rare-label weights are clipped so rarest techniques cannot dominate the gradient.
  • Training setup: The implementation fine-tunes roberta-base for 40 epochs with AdamW, a learning rate of 1 × 10−5, and random seed 42.Training uses a linear learning-rate schedule, Hugging Face Trainer, and CodeCarbon instrumentation.
  • Evaluation protocol: Checkpoint selection maximises macro-F1 on an evaluation split, but the choice of split materially affects validity and is corrected later.The passage explicitly identifies split choice as a major issue analysed in section 6.

5 Evaluation Protocol and Baselines

The fine-tuned classifier substantially outperforms the zero-shot similarity ranker on ranking metrics, despite using only about 1,086 gold examples. Its remaining weakness is rare-technique performance, and corrected-protocol results are lower than the pre-correction single-run figures.

  • Baseline comparison: Recall@5 rises from 0.322 to 0.686, roughly doubling under fine-tuning versus the zero-shot similarity baseline.The classifier improves across every reported ranking metric, with MRR increasing from 0.397 to 0.620.
  • Baseline comparison: 1,086 gold examples provide clear gains despite the limited training set.After the in-vocabulary filter, the training set contains 1,083 examples.
  • Baseline comparison: Macro-F1 is about 0.20, indicating the remaining weakness is rare-technique performance.This limitation motivates the subsequent label-expansion study.
  • Evaluation protocol: 0.673 ± 0.019 is the corrected-protocol recall@5 for the five-seed gold-only model, versus the pre-correction single-run figure.The corrected-protocol macro-F1 is 0.177 ± 0.014.

6 LLM-Assisted Label Expansion

The section evaluates LLM-generated ATT&CK labels as a scalable alternative to expert curation, finding that apparent gains vanish under replication and corrected checkpoint selection. Expert-curated data improves consistently with scale, whereas LLM-labeled expansion does not.

  • Labeler design: The labeler maps each CVE into exploitation, primary-impact, and secondary-impact slots, with local Ollama and Anthropic Claude backends plus optional assertive prompting and two-step reasoning.The assertive prompt targets recall by requiring supported assignments, while the two-step procedure was tested as an optional alternative.
  • Labeler benchmark: 0.392 micro-F1: the best configuration, qwen3.5:122b with a single assertive call, nearly matches the classifier reference of 0.407 on the full split.Model capacity mattered more than prompt engineering: increasing from 35B to 122B raised probe micro-F1 by 0.13, while assertive prompting added only 0.02.
  • Initial expansion results: A single fixed-seed run made expansion appear harmful, with recall@5 decreasing by 0.050 and macro-F1 by 0.021.This result was later shown to be unreliable because repeated runs disagreed substantially.
  • Initial expansion results: Across five seeds, the union model instead improved recall@3 by +0.038, recall@5 by +0.030, and micro-F1 by +0.020, while macro-F1 stayed flat.The apparent reversal motivated independent replication and a scaling sweep.
  • Corrected evaluation: Under corrected validation-split checkpoint selection, expansion failed to improve ranking: recall@5 fell from 0.673 to 0.655 with 297 rows and 0.651 with 984 rows.The original protocol selected the best macro-F1 among 40 checkpoints on a 119-example test split, amplifying evaluation noise.
  • Scaling study: Gold-only training improved monotonically with trusted-data size, raising recall@5 from 0.556 to 0.673 and macro-F1 from 0.114 to 0.177.The vocabulary and filtered test set were frozen to the full-gold version so only training size varied.

7 Discussion and Limitations

The study’s low absolute scores reflect a genuinely difficult, subjective mapping task with scarce and imperfect ground truth. Its conclusions are constrained by a small, biased dataset, evolving ATT&CK schemas, and limited evidence about expansion and encoder choices.

  • Task difficulty: CVE-to-ATT&CK mapping is genuinely difficult and partly subjective because vulnerability descriptions characterize flaws while techniques characterize attacker behavior.Even expert analysts can disagree on mappings, so low absolute F1 should not be compared with near-ceiling results from well-posed classification tasks.
  • Dataset limitations: The ∼1,200-CVE gold set over-represents vulnerabilities exploited in the wild, especially remote server exploitation, limiting transfer to never-exploited CVEs.Its small size also restricts the technique vocabulary by excluding techniques with too few examples.
  • Ground truth: Public ground-truth collections are scarce, and CWE→CAPEC→ATT&CK-derived evaluations can reward reproducing table-expansion artifacts rather than analyst-relevant mappings.Expert records are not necessarily exhaustive: omitted techniques are not established true negatives when descriptions support multiple plausible paths.
  • Version and schema drift: ATT&CK version and schema drift can change techniques, sub-techniques, names, tactic associations, and STIX relationships after the original mappings were authored.The 2021 CTID mappings targeted an older ATT&CK release, so consumers must track the evolving ontology and Enterprise schema.
  • Expansion scope: At ≈0.39 agreement and up to 984 added CVEs, LLM expansion showed no reliable gain and reduced rare-technique coverage, but this verdict is limited to one model family, prompt, and five seeds.The result rules out scale alone rescuing labels at this agreement level, not expansion under different configurations or higher-quality labels.
  • Model choice: The encoder study used roberta-base without systematic base-model search, leaving domain-adapted and larger encoders for future work while indicating label quality is the bottleneck.On ModernBERT, the expansion degradation replicated, and its gold-only ranking did not improve over gold-only roberta-base.

8 Conclusion

The conclusion highlights that a curated gold-set classifier substantially outperforms zero-shot similarity, while stricter evaluation overturns inconsistent findings about LLM-assisted expansion. It also identifies higher-agreement labeling, curated-data scaling, and source-level inputs as directions for improvement.

  • Main findings: A supervised classifier trained on a small curated gold set roughly doubles recall@5 versus a zero-shot similarity baseline and improves every ranking metric.The conclusion presents trustworthy expert labels as effective even at modest scale.
  • Evaluation conclusions: One seed found expansion harmful, five seeds found it helpful, and independent replication with a size sweep found no effect.A final rerun under the corrected protocol confirmed the null result.
  • Evaluation conclusions: Selecting the best of 40 checkpoints on a 119-example test split made evaluation noise a central source of inconsistent conclusions.The conclusion distinguishes this mechanism from seed variance alone.
  • Future work: Higher-agreement labeling aimed at rare techniques and enlarging the curated gold set are identified as the clearest technical opportunities.Suggested labeling approaches include human-reviewed silver labels and retaining only high-confidence slots.
  • Future work: A multi-input classifier could combine commit messages, changed code, and advisory text to rank ATT&CK techniques with evidence linked to relevant source lines.This direction targets information omitted from short CVE descriptions and could support end-to-end draft vulnerability-record generation.

Funding

The work was co-funded by CIRCL and the European Union through the AIPITCH project, with EU support provided under grant agreement No 101190545.

  • Funding: The project received co-funding from CIRCL and the European Union via AIPITCH, supported by grant agreement No 101190545.The authors state that the expressed views are their own and do not necessarily reflect those of the European Union or its cybersecurity competence centre.
Loading 2607.25572v1…