PaperlayerView exact revision on arXiv

Source-linked AI summary

How Reliable Are NVD CWE Labels? A Large-Scale Semantic Audit with Seclometry

Yu Nong, Yao Du, Majid Behravan, Haipeng Cai

arXiv:2608.21977v1cs.CRcs.SE

TL;DR

NVD CWE labels are widely used as ground truth, but their semantic reliability had not been measured at scale. This paper introduces and validates CWEAGENT, a code-semantics-grounded auditing instrument, then applies it to 15,556 open-source CVEs to quantify label quality and characterize recurring mislabels.

  • Problem

    The semantic consistency of NVD CWE labels with the weakness supported by CVE evidence had not been systematically measured, although these labels underpin downstream vulnerability datasets and tools.

  • Method

    CWEAGENT uses seclometry to reason over advisory, patch, and code evidence, constructs CWE knowledge independently from MITRE definitions, and separates CWE prediction from label adjudication.

  • Results

    49.70% of NVD labels exactly matched the code-grounded CWE, 31.37% were defensible alternatives or ambiguity cases, and 3.63% were evidence-inconsistent.

  • Takeaways & Limitations

    CWE noise is structured across weakness types and assigning organizations, with recurring semantic failure modes documented through 434 manually confirmed NVD mislabels.

  • Takeaways & Limitations

    Existing automated classifiers are limited as audit instruments because description-level cues, NVD-label training, frequent-class coverage, and black-box predictions hinder semantic label measurement.

Abstract

from arXiv · show

CWE labels in the National Vulnerability Database (NVD) are widely treated as ground truth for vulnerability search, scanner evaluation, benchmark construction, learning-based security tools, and vulnerability prioritization. Yet their reliability has not been systematically measured at scale, despite growing concerns about NVD's enrichment backlog and anecdotal reports of inaccurate, ambiguous, or missing labels. This paper presents a large-scale, code-semantics-grounded measurement of CWE labeling quality in NVD. We build CWEAgent, a validated auditing instrument based on seclometry, a structured representation of vulnerability semantics that captures the root cause, trigger condition, violated security property, exploit mechanism, and impact of vulnerable code. On a manually curated benchmark of 100 open-source CVEs, CWEAgent achieves 85% top-1 accuracy and 92% ambiguity-aware accuracy. Applying CWEAgent to 15,556 open-source CVEs disclosed from 2017-2026, we find that only 49.70% of NVD CWE labels exactly match the code-grounded label. Another 31.37% are defensible alternatives under taxonomy ambiguity, while 3.63% are evidence-inconsistent likely errors. Label reliability varies sharply by assigning organization and weakness type, and apparent project- or language-level differences are largely composition effects of those underlying weakness types. Evidence-inconsistent labels have also increased over time. Through manual review of 434 confirmed mislabels, we identify six recurring error patterns, showing that CWE noise is a structural problem in vulnerability metadata rather than isolated annotation mistakes.

1. Introduction

This paper measures the semantic reliability of NVD CWE labels at scale using a code-semantics-grounded auditing instrument designed to distinguish exact matches, defensible disagreement, and evidence-inconsistent errors. Across open-source CVEs, it finds substantial label noise, structured variation by weakness type and assigning organization, and recurring mislabeling patterns.

  • NVD CWE labels support vulnerability search, benchmarking, learning-based tools, prioritization, and remediation reasoning, so label noise can propagate into downstream datasets and decisions.
  • CWE label reliability had not been systematically measured against the weakness semantics supported by each CVE’s advisory, patch, and code evidence.
  • CWEAGENT combines seclometry-based evidence reasoning with separate prediction and adjudication steps, avoiding training on NVD labels and distinguishing defensible alternatives from errors.
  • 85% top-1 accuracy and 92% ambiguity-aware accuracy were achieved on a manually curated benchmark of 100 CVEs, while the Arbitrator reached 90% overall accuracy on 100 adjudication cases.
  • 49.70% of NVD labels exactly matched the code-grounded CWE, 31.37% were defensible alternatives or overlap-ambiguous, and 3.63% were evidence-inconsistent.
  • Evidence-inconsistent labels rose from roughly 1–3% in 2017–2018 to 3–6% in 2021–2026 despite no clear monotonic trend in strict exact-match rates.
  • Label quality varies sharply by weakness type and assigning organization, while apparent project- and language-level differences are largely composition effects.
  • Manual review confirmed 434 NVD mislabels among 564 flagged records, identifying six recurring semantic failure patterns.

2. Background and Motivation

CWE assignment should identify the specific root-cause weakness rather than a vulnerability’s consequence or symptom. Existing classifiers and metadata practices are insufficient for auditing this distinction, motivating CWEAGENT’s evidence-grounded design.

  • MITRE guidance favors the most specific applicable root-cause CWE and marks entries as allowed, discouraged, or prohibited for direct mapping.
  • Consequences such as disclosure, privilege gain, authentication bypass, or code execution do not necessarily identify the mechanism that makes the vulnerability possible.
  • In CVE-2026-26014, nonce reuse is the root cause mapped to CWE-323, whereas key-material exposure is a consequence and NVD assigned CWE-200.
  • This consequence-versus-root-cause mismatch can cause CWE-323 searches to miss the CVE and CWE-200 analyses to include it incorrectly.
  • Prior metadata studies identified NVD quality issues and invalid mappings but did not measure whether assigned labels were semantically consistent with CVE evidence.
  • Existing classifiers rely on descriptions, NVD labels, frequent classes, or black-box predictions, limiting their ability to audit long-tail labels and distinguish ambiguity from mislabeling.
  • CWEAGENT addresses these gaps by constructing CWE knowledge from MITRE, reasoning over advisory, patch, and code evidence, and separating prediction from adjudication.

3. Design of CWEAGENT

CWEAGENT separates evidence-grounded CWE prediction from adjudication of the existing NVD label. Its seclometry representation aligns CWE definitions and CVE evidence across structured security-semantic fields, enabling candidate narrowing, scoring, and reliability judgments.

  • Design overview: CWEAGENT compares NVD labels against code-supported weakness mechanisms, distinguishing exact matches, defensible alternatives, ambiguity, and evidence-inconsistent labels.The Classifier predicts an evidence-supported CWE, while the Arbitrator determines the status of the NVD label.
  • Classifier workflow: Phase 1 builds reusable CWE seclometries and a multi-view cluster index; Phase 2 gathers advisory, patch, code, and static-analysis evidence before narrowing and scoring candidates.The Classifier’s two-phase workflow keeps catalog-wide reasoning offline and reuses it during per-CVE inference.
  • Seclometry: Seclometry represents CWE definitions and CVE evidence uniformly through root cause, trigger conditions, violated invariants, exploit mechanisms, impacts, and related security-semantic fields.This common representation reduces reliance on surface-level consequence words when matching across different text genres.
  • Seclometry: Each seclometry combines an eight-field natural-language security signature with a normalized categorical schema for reasoning, clustering, candidate filtering, and scoring.The same two-component format is used for 842 ALLOWED CWEs and for CVE seclometries.
  • Seclometry: Key Invariant Violated and Non-Examples/Boundaries distinguish neighboring weaknesses that share broad mechanisms or impacts but differ in defining conditions.For example, CWE-787 and CWE-125 both involve out-of-bounds access, but differ between invalid writes and reads.

4. CWEAGENT Implementation

CWEAGENT assigns different LLMs to catalog construction, scalable per-CVE inference, and adjudication. Fixed prompts, model versions, and temperature 0 make calls deterministic, while evidence refinement is bounded to three iterations.

  • Model assignment: Phase 1 uses Claude 4.6 Opus for one-time seclometry generation and multi-view clustering, prioritizing catalog-wide reasoning quality.This phase constructs reusable CWE-side knowledge before per-CVE measurement.
  • Model assignment: Phase 2 uses GPT-5.4-mini for the Seclometry Builder and CWE Matcher, which run once per CVE and must scale to wild-scan measurement.The Arbitrator uses Claude Sonnet 4.6, separating its model family from the Phase 2 Classifier.
  • Determinism: Fixed prompts, fixed model versions, and temperature 0 make each LLM call deterministic for a given input.Post-processing repairs output-format problems without changing semantic decisions.
  • Evidence refinement: The evidence-refinement loop between Agents 1 and 2 is capped at three iterations because further iterations rarely change the predicted CWE.The budget was selected empirically.
  • Evidence collection: The Seclometry Builder retrieves advisories and source artifacts and invokes CodeQL query packs for eight supported programming languages.Supported languages are C/C++, Python, JavaScript, Java, C#, Go, and Ruby.

5. Evaluation

The evaluation separates CWE prediction from label adjudication and tests both against independently curated evidence. CWEAgent performs strongly on controlled prediction and adjudication tasks, supporting its use for large-scale measurement.

  • Evaluation: CWEAgent’s Classifier and Arbitrator are evaluated separately because prediction accuracy and label-adjudication reliability require different ground truths.The Classifier is tested where the correct CWE is known, while the Arbitrator is tested where existing-label correctness is known.
  • Classifier Prediction Accuracy (RQ1): 100 CVEs form the manually curated prediction benchmark, spanning 38 CWEs across open-source projects from 2017–2026.The benchmark retains only cases with clearly correct NVD labels and uses both exact-match and ambiguity-aware accuracy.
  • Classifier Prediction Accuracy (RQ1): 85% top-1 exact-match and 92% ambiguity-aware accuracy are achieved by CWEAgent on the controlled benchmark.The system outperforms the evaluated single-prompt LLM baselines and TreeVul.
  • Arbitrator Adjudication Reliability (RQ2): The Arbitrator is evaluated on 100 cases combining 50 confirmed NVD mislabels and 50 confirmed-correct labels.Independent manual adjudication assigns each case to count as correct, discard, or keep as incorrect, with Cohen’s κ = 0.84.
  • Arbitrator Adjudication Reliability (RQ2): 90% overall accuracy is achieved by the Arbitrator, with per-category F1 scores between 89% and 91% and precision and recall of at least 82%.The clustered scores indicate balanced reliability rather than dominance by a single decision category.

6. Measuring CWE Mislabeling in the NVD

The large-scale measurement finds that NVD labels are frequently imprecise but less often evidence-inconsistent, with reliability shaped mainly by weakness type and assigning organization. Project and language differences largely reflect the composition of their vulnerabilities.

  • Measurement Scope: 15,556 CVEs with GitHub fix commits are adjudicated by comparing CWEAgent predictions with current NVD labels.The corpus covers CVEs disclosed from 2017 through 2026.
  • Overall Prevalence: 49.70% of NVD labels exactly match, while 25.01% are overlap ambiguities and 6.36% are defensible alternatives, yielding 81.07% ambiguity-aware correctness.3.63% are evidence-inconsistent verdicts and 15.30% are classifier errors where NVD is correct.
  • Longitudinal Trend: Evidence-inconsistent labels rise from 1–3% in 2017–2018 to 3–6% in 2021–2026, with 2024–2026 contributing 228 such labels.That recent cohort accounts for over 40% of all evidence-inconsistent labels in the corpus.
  • Distribution by Weakness Type: Exact-match varies widely across weakness families and specific CWEs, whereas evidence-inconsistency remains comparatively narrow and localizes to particular weakness types.Heap-based buffer overflow (CWE-122) reaches 11.94% evidence-inconsistency, over triple the corpus rate.
  • Distribution by Organization, Project, and Language: Assigning CNAs produce roughly a 70-point exact-match spread, while their evidence-inconsistency spread is narrower.Project- and language-level differences are largely composition effects, and Radare2 is an exception with a 15.49% S4 rate.

6.2. Severity and Impact

CVSS severity and exploitation status do not predict CWE label trustworthiness. High-severity and exploited-vulnerability subsets retain label-quality rates close to the broader corpus.

  • CVSS Severity: CVSS severity does not predict CWE label quality: exact-match spans 54.46% to 48.58%, while evidence-inconsistency spans 3.10% to 3.78%.Neither measure varies monotonically with severity, and Critical CVEs are no more accurately labeled than Medium CVEs.
  • CVSS Severity: Filtering by severity does not materially improve CWE-label reliability, although it changes a dataset’s risk profile.High-severity subsets retain nearly the same ambiguity-aware correctness and evidence-inconsistency rates as lower-severity subsets.
  • Exploitation Status: Among 241 Exploit-DB CVEs, evidence-inconsistency is 3.73% and exact-match is 56.85%, close to the corpus-wide rates.The 39 KEV CVEs have a 2.56% S4 rate, too few for a standalone rate but consistent with the same pattern.
  • Conclusion: Neither CVSS severity nor real-world exploitation predicts CWE label quality, so impact is not a proxy for label trustworthiness.The paper instead recommends controls based on weakness semantics, evidence provenance, and assigning organization.

6.3. Failure Patterns

Manual review shows that confirmed NVD mislabels follow six recurring semantic patterns, usually arising from relationships between causes, effects, sibling weaknesses, or taxonomy branches. The errors are diffuse across CWE pairs, with most crossing weakness families.

  • Review Method: 434 of 564 evidence-inconsistent records are confirmed as genuine NVD mislabels through independent review of primary evidence.The review excludes defensible-but-imprecise readings and reports the confirmed set to NVD.
  • Failure Patterns: The six recurring patterns are consequence-versus-root-cause, sibling sub-type, injection sink/interpreter, access-control and identity, discouraged or wrong-branch, and multi-CWE accumulation.The top three patterns cover nearly two-thirds of confirmed mislabels.
  • Consequence vs. Root Cause: 23.1% of confirmed mislabels confuse a downstream consequence with the root cause, such as labeling nonce reuse as information exposure.Consequence CWEs can fit an end state even when the description names the mechanism.
  • Subtype Mislabel: 21.3% select the wrong sibling subtype within a family, including an out-of-bounds read labeled as an out-of-bounds write.CWE normalization can map memory-buffer weaknesses toward a coarser parent and obscure the read/write distinction.
  • Injection and Access-Control Failures: 19.5% confuse injection sinks or interpreters, while 15.3% conflate access-control and identity failures.Examples include Python eval() labeled as data-query injection and improper identity binding labeled as missing authentication.
  • Taxonomy and Record-Level Failures: 10.3% use discouraged or wrong-branch labels, and 7.8% retain unsupported labels alongside correct multi-CWE assignments.Examples include prohibited CWE-840 for an input-validation flaw and unsupported CWE-77 alongside XSS labels.
  • CWE Migration: 80.5% of confirmed mislabels cross weakness families, while 384 distinct NVD-to-correct-CWE pairs show that mislabeling is diffuse rather than a few recurring swaps.The only recurring swap highlighted is CWE-787 to CWE-125 for out-of-bounds write versus read.

6.4. Other Bad Labeling Practices

NVD contains many labels that are defensible but not maximally precise, alongside widespread use of discouraged or prohibited CWEs. These practices shift weakness resolution onto downstream consumers without necessarily making labels outright wrong.

  • Ambiguous labels: Roughly 31% of the corpus are ambiguous labels that remain defensible but are not the most-specific assignments supported by the evidence.These cases reflect genuine overlap in the hierarchical CWE taxonomy rather than labeler errors.
  • Discouraged and prohibited labels: Over 16% of CWE assignments use CWEs that MITRE discourages or prohibits for direct CVE mapping.The corpus contains 2,536 such CVEs.
  • Discouraged and prohibited labels: CWE-20, CWE-200, CWE-400, CWE-284, and CWE-119 are common abstract CWEs that cannot be the most-specific labels for evidence-supported weaknesses.Their S4 rates are at or below the corpus average, so they are seldom wrong but maximally imprecise.
  • Downstream cost: Both ambiguous labels and discouraged or prohibited CWEs force consumers to recover the actual weakness from advisories, patches, or code.The cost is reduced precision rather than necessarily outright inconsistency.

6.5. Ecosystem Side Effects

CWE mislabeling can arise from source disagreement, later enrichment revisions, or incomplete source reports. These cases make provenance and reconciliation central to understanding how incorrect labels enter NVD records.

  • Across sources: A correct CWE can already be present on a CVE when a disagreeing CNA or NVD label is also recorded.This is a reconciliation failure, and it connects to the assigning CNA’s role as the strongest metadata-level signal of label quality.
  • Across labeling history: Later NVD enrichment can add an unsupported CWE after a correct label was present at publication.For CVE-2024-42477, CWE-125 was present at publication and later enrichment added unsupported CWE-401.
  • Upstream source reports: An omitted mechanism in a source report can cause a labeler to assign the wrong weakness even when the label follows the report faithfully.For CVE-2022-1160, the description said “heap buffer overflow,” while sanitizer output and the fixing commit revealed an out-of-bounds read.
  • Ecosystem side effects: Three ecosystem failure modes involve conflicting source labels, incorrect labels added during later enrichment, and source reports that omit the mechanism.In each case, the issue concerns reconciliation or provenance rather than missing information.

7. Discussion

NVD CWE data has a large precision problem but a smaller correctness problem: many non-exact labels are defensible, while a smaller tail contradicts the available evidence. Provenance and patch-grounded auditing offer practical ways to interpret and improve use of the data, within the study’s scope and validity limits.

  • Implications for using NVD CWE data: Roughly a third of the corpus contains defensible but non-exact labels, whereas only 3.63% are evidence-inconsistent likely errors.Treating all disagreement as incorrect would conflate taxonomy imprecision with correctness failures.
  • Implications for using NVD CWE data: Labels are most trustworthy at the weakness-family level and least reliable for specific sub-types such as exact read/write operations.Evidence inconsistency is low and roughly flat across families, while precision varies widely within families.
  • Implications for using NVD CWE data: Assigning CNA is a practical confidence signal because label quality varies more across CNAs than across intrinsic vulnerability properties such as severity.Consumers can weight or flag labels by provenance using metadata already present in NVD entries.
  • Patch-grounded auditing: Patch-grounded auditing can recover sub-type errors, consequence-for-cause cases, and unsupported co-assigned labels that description-driven labeling misses.The paper presents this as scalable auditing rather than a replacement for human labeling.
  • Application to mitigating labeling backlog: The backlog application isolates the CWE-labeling subtask and does not model the full NVD enrichment process.The study presents CWEAGENT as an enrichment aid while retaining human oversight for final NVD-quality enrichment.
  • Threats to validity: The reported rates may not generalize beyond open-source CVEs with locatable fixing commits, and S4 estimates depend partly on CWEAGENT’s judgments.The study mitigates instrument-validity concerns through separate evaluation and manual review of all S4 cases.

8. Related Work

Prior work predicts CWE labels or detects administratively invalid mappings, but this paper addresses whether assigned labels are semantically consistent with advisory, patch, and code evidence. CWEAGENT combines LLM-based semantic extraction, CWE matching, and adjudication for that auditing role.

  • CWE label prediction: Existing CWE classifiers predict labels from descriptions, code, fixing diffs, or CWE-catalog structure, whereas this paper measures assigned-label quality.The distinction is between generating likely labels and auditing whether NVD labels fit the available evidence.
  • NVD and CWE label quality: Prior NVD-quality studies identify missing, inconsistent, or rule-invalid metadata, but do not generally test whether administratively valid labels contradict code-level evidence.The paper frames this as a semantic problem requiring evidence-grounded adjudication.
  • LLM agents for security analysis: CWEAGENT uses LLMs to compose semantic extraction, full-catalog CWE matching, and label adjudication into an auditing instrument.Its measurement role differs from LLM applications for vulnerability detection, localization, repair, or generic security reasoning.

9. Conclusion

CWEAGENT audits NVD CWE labels against weakness mechanisms supported by advisory, patch, and code evidence. Across 15,556 open-source CVEs, labels are usually defensible but often imprecise, with evidence-inconsistent cases comprising 3.63%.

  • CWEAGENT compares each NVD CWE label with the weakness supported by advisory, patch, and code evidence.
  • 15,556 open-source CVEs were measured, showing that NVD labels are usually defensible but often imprecise.
  • 3.63% of NVD labels were evidence-inconsistent, while quality tracked the assigning CNA and weakness type rather than severity.
Loading 2608.21977v1…