PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the NIST Bugs Framework Against CWE as a Successor for Automated Vulnerability Classification

Md Nazmul Hoque, Shaswata Mitra, Subash Neupane, Sudip Mittal, Shahram Rahimi

arXiv:2609.16433v1cs.CRcs.SE

TL;DR

Overlapping CWE entries allow the same vulnerability to receive multiple defensible labels, while evidence that BF addresses automation’s classification challenges has been limited. This paper evaluates BF through a screened literature corpus, specification analysis, and reproducibility studies; its findings characterize BF as more structured and automation-suitable, while identifying limits and areas for further work.

  • Problem

    CWE’s overlapping entries permit multiple defensible labels for one vulnerability, and BF had not yet been empirically evaluated against challenges to automated classification.

    Source: §1 · p. 1, §3.2 · p. 7

  • Method

    The paper combines a screened review of CVE-to-CWE research, analysis of BF specification correspondence, and inter-rater and automated determinism studies of CVE-to-BF classification.

    Source: §2 · p. 4, §8.3 · p. 27

  • Results

    The findings position BF as a more structured and automation-suitable target, while automated BF chains do not always match published verified chains.

    Source: §10 · p. 36

  • Takeaways & Limitations

    The authors identify per-axis validation, annotation consistency, and alignment with verified chains as priorities for future work.

    Source: §10 · p. 36

    The four-to-four mapping and case-study labels use the specification snapshot at writing, whereas a proposed system would consume the live BF taxonomy.

    Source: §9 · p. 36

Abstract

from arXiv · show

Vulnerability classification based on root cause weaknesses is essential for numerous cybersecurity activities, where the Common Weakness Enumeration (CWE) serves as a public repository of such flaws. However, its overlapping entries create a non-orthogonal structure. The result is the same vulnerability being mapped to multiple weaknesses, complicating Root Cause Analysis (RCA) and triage. To address this, NIST Special Publication 800-231 introduces the Bugs Framework (BF), which organizes vulnerabilities into <cause, operation, consequence> triples and links such triples into a causal chain, so that a vulnerability carries its root cause and its sink together instead of a single terminal label. To date, however, BF has been specified but not evaluated regarding its performance against the challenges to automated classification. The evidence required for adoption has not been investigated empirically. We evaluate BF as a classification target and a complement to CWE using a systematically screened corpus of automated Common Vulnerabilities and Exposures (CVEs) linked to CWE research. We assess the reproducibility of CVE-to-BF classification through two evaluations. The first is qualitative: an anonymized inter-rater study in which 2 subject-matter experts (SMEs) independently mapped 13 CVEs onto the four BF axes. Annotators showed strong agreement on the cause and operation axes, while the attribute axis indicated fair agreement. We also tested our automated framework across two large language model (LLM) deployments under different budgets for reproducibility analysis. Despite limitations, such as evidence availability and the absence of retrievable fix commits for closed-source software, our findings support the claim that BF is a more structured and automation-friendly framework than CWE. Our exploration reveals specific gaps in BF, including under-specified guidance on attributes.

1 Introduction

CWE’s overlapping entries let the same vulnerability receive multiple defensible labels, while sink-level labeling can obscure root cause. The paper evaluates NIST’s Bugs Framework as a structured alternative and complement, combining a literature-grounded analysis with reproducibility studies and case studies.

1 Introduction

  • CWE’s overlapping entries and differing abstraction levels allow multiple defensible labels for one CVE, weakening classification and obscuring remediation priorities.The introduction frames this as a central problem for trustworthy automation.Source: §1 · p. 2
  • BF represents a weakness through a cause, operation, and consequence, with classes organized around disjoint operation sets.The introduction presents this structure as an alternative to a flat sink-level CWE label.Source: Figure 1 · p. 2, §1 · p. 3
  • The paper asks how recurring structural limits in automated CVE-to-CWE classification can be addressed and whether BF addresses them by design.It also asks whether the correspondence can be traced to BF specification language and published chains, and how reproducible CVE-to-BF classification is in practice.Source: §1 · p. 3
  • The study combines a systematic literature review, specification-level correspondence analysis, and two empirical evaluations of BF classification.The evaluations comprise an anonymous inter-rater reliability study and an automated determinism study across two deployments and budget combinations.Source: §1 · p. 3
  • The authors characterize a screened corpus of automated CVE-to-CWE research and analyze four recurring CWE failures: non-orthogonality, intractable target space, sink-only labeling, and no causal chain.They report that surveyed systems repeatedly accommodate these failures rather than overcome them.Source: §1 · p. 3
  • Case studies connect documented CWE assignments and reassignments to the structural failures they expose and to BF properties that address them.The introduction also identifies the operational stakes of classification quality, citing the Equifax breach as an example.Source: §1 · pp. 2–3

2 Methodology

The methodology follows a distinct procedure for each research question: a systematic corpus review and characterization, a specification correspondence analysis, and two empirical reproducibility studies. It uses predefined screening and extraction procedures and a five-axis schema to support synthesis of the included research.

2 Methodology

  • The study pairs a systematic review and specification-level analysis with anonymous inter-rater and automated determinism studies using a curated, evidence-based pipeline.The procedure is organized around the three research questions.Source: §2 · p. 4

2.1 Corpus Collection: A PRISMA Procedure

  • The PRISMA-style corpus procedure sources research across six databases, screens records for eligibility, and synthesizes the retained papers by method family.Twenty papers met the stated inclusion criteria, including requirements on CVE inputs, CWE outputs, automated methods, corpus size, and traceable evaluation.Source: §2.1 · pp. 4–5, Figure 2 · p. 4

2.2 Manual Data Extraction

  • Authors screened records and extracted eligible-paper properties with a predefined form, then checked a sample of records against the original papers.Discrepancies in the checked sample were resolved through joint review.Source: §2.2 · p. 5

2.3 Characterization Schema

  • The five-axis characterization schema records input artifact, target form, method family, evaluation setting, and stated limitation, and supports synthesis of recurring limitation patterns.The integrated tables report performance alongside test-set conditions because results depend on label-space size, class composition, and evaluation metric.Source: §2.3 · p. 5

3 Preliminaries

This section defines four structural failures in CVE–CWE classification and contrasts them with BF’s structured, multidimensional representation of weaknesses and causal chains. It illustrates CWE’s multiple defensible mappings and situates BF within its research program.

3.1 Operational Definitions

  • Orthogonality requires non-overlapping categories, while determinism requires independent qualified experts to converge on the same category above an agreed threshold.BF’s disjoint operation sets are intended to provide orthogonality.Source: §3.1 · p. 6
  • CWE’s four structural failures are overlapping hierarchy (F1), a skewed target space (F2), sink-only labels (F3), and unordered weakness sets without per-instance causal chains (F4).The paper defines structural inconsistency as the simultaneous presence of F1–F4 in the taxonomy and mapping procedure, while noting that per-CVE NVD records encode neither root-cause mapping nor named weakness chains.Source: §3.1 · pp. 6–7

3.2 Demonstration of Multiple Defensible CWE Mappings

  • CVE-2021-3156 admits several defensible CWE labels, including CWE-193, CWE-122, CWE-119, CWE-787, and CWE-120.These assignments span different descriptions and abstraction levels of the same defect.Source: §3.2 · p. 7

3.3 Overview of the Bugs Framework (BF)

  • BF represents a weakness as a cause–operation–consequence triple with attributes, and a vulnerability as a causal chain of weaknesses ending in a security failure.Its specification also requires disjoint operation sets across classes; this study uses a subset of the specified taxonomy.Source: §3.3 · pp. 7–9, Figure 4 · p. 9

3.4 The BF Research Program as a Design Response

  • The BF research program includes the formal specification, the antecedent triple-based weakness model, and a separate metric-design contribution.The antecedent BF paper establishes the cause–operation-to-consequence triple as the unit of weakness specification.Source: §3.4 · p. 9

4 CTI Frameworks and Defensive Operations Pipeline

The CTI pipeline connects vulnerability disclosure and classification to adversary modeling and defensive countermeasures. Because downstream mappings depend on upstream labels, ambiguity in CVE-to-CWE classification constrains the precision of later artifacts.

4 CTI Frameworks and Defensive Operations Pipeline

  • The CTI pipeline links a CVE record and NVD enrichment to later adversary and defensive frameworks, with each stage bounded by the fidelity of the preceding one.The depicted flow runs from CVE disclosure to a D3FEND countermeasure.Source: §4 · pp. 9–10, Figure 5 · p. 10

4.1 Operational Workflow of the CTI Pipeline

  • Enterprise workflows add SSVC remediation priorities and EPSS exploitation probabilities alongside CWE labels and CVSS severity scores.CAPEC maps weaknesses to attacker methodologies, while ATT&CK supplies operational tactics, techniques, and procedures.Source: §4.1 · p. 10
  • ATT&CK is coarser-grained than CWE and defers software attack-vector coverage to CAPEC and CWE.A single ATT&CK technique can encompass vulnerabilities with distinct underlying weaknesses.Source: §4.1 · p. 10

4.2 Bidirectional Mapping between ATT&CK and D3FEND

  • D3FEND’s defensive recommendations can be no more specific than upstream CWE and CAPEC labels, so classification ambiguity carries into later mappings.Replacing a CWE label with a BF causal chain removes this bound at its source.Source: §4.2 · pp. 10–11

5 CVE-to-CWE Automation Corpus

The 20-paper CVE-to-CWE corpus shows that reported performance depends on target-taxonomy structure, with systems adapting through restricted label spaces, alternative prediction forms, label repair, and LLMs. Its historical progression shows that more capable systems redirect effort to mitigations rather than dissolving CWE’s structural failures.

5.1 Methods and Reported Performance

  • Reported accuracy rises as the label space narrows and falls when the long tail is admitted, regardless of model family.Figure 6 presents this pattern as a conceptual performance ceiling created by CWE label-space ambiguity.Source: §5.1 · pp. 11–12, Figure 6 · p. 11
  • The corpus tables pair methodological approaches with headline performance and BF-versus-CWE relevance, reporting test-set conditions alongside results.Table 2a covers model family, training regime, dataset, and label-space size; Table 2b covers headline performance and framework relevance.Source: Table 2a · p. 12, Table 2b · p. 14, §5.1 · p. 14

5.2 Four Adaptive Strategies for Thematic Synthesis

  • Some systems restrict the label space, including top-ten, 25-class, 57-class, 160-class, and 205-class views.The Temporal LR baseline also reinforces the intractability reading: a skewed label distribution sets a hard performance ceiling even with temporal validation.Source: §5.2 · p. 14
  • Other systems change prediction form through hierarchical or top-K prediction, retrieval, cross-taxonomy prediction, or graph relation prediction.VulnScopper [4] reports that 44.3% of CVEs cataloged in two databases carry a different CWE, attributing the divergence to the hierarchy permitting general or specific labels for the same defect.Source: §5.2 · p. 14
  • Repair systems target invalid or insufficiently specific NVD mappings; FixV2W [74] recovers prohibited mappings within the top ten in 65% of cases.More than half (55%) of NVD CVEs carry invalid or insufficiently detailed CWE mappings.Source: §5.2 · pp. 14–15
  • LLM-based systems use root-cause term extraction or direct classification, but LLM-VulnClass reports a simple TF-IDF baseline outperforming LLM embeddings on the same task.VWC-BERT also shows that upstream CWE uncertainty propagates into downstream CWE-to-CAPEC mapping.Source: §5.2 · p. 15

5.3 Historical Progression

  • Chronologically, systems redirect effort to narrower targets, hierarchical ranking, label repair, and LLM term extraction rather than dissolving the observed failures.Figure 7 renders this progression as thematic clustering across the five Characterization Schema axes.Source: Figure 7 · p. 15, §5.3 · p. 16

6 Comparative Analysis of the Bugs Framework Against CWE’s Structural Failures

The analysis maps each of four operational CWE failures to a BF design property, arguing that BF addresses them by construction. The mapping reduces label overlap and preserves causes and chains, but does not eliminate the learning challenge of predicting full combinations.

6.1 Property-by-Failure Mapping

  • BF properties correspond one-to-one with the four operational CWE failures, while the inter-rater results provide partial confirmation on cause and operation axes.Figure 8 records the failure-to-property matrix; Figure 10 shows agreement across 13 CVEs and 52 axis cells.Source: §6.1 · p. 16, Figure 8 · p. 17, §6.1.4 · p. 19

6.1.1 Orthogonal weakness classes address the non-orthogonal hierarchy (F1).

  • Disjoint operation sets make BF weakness classes orthogonal, removing competing ancestor, descendant, or sibling labels for a defect.This design does not itself guarantee that an automated system selects the correct class.Source: §6.1.1 · p. 16

6.1.2 Structured composition over a small class set addresses the intractable target space (F2).

  • BF composes weaknesses from small per-axis class sets, reducing the per-axis target space while leaving a large, skewed space of valid full combinations.The study reports κ=0.876 for cause, κ=0.900 for operation, and κ=0.235 for attribute; the attribute set is largest and most ambiguous.Source: §6.1.2 · pp. 16–18

6.1.3 Explicit cause and operation axes address sink-only labeling (F3).

  • Separate cause and operation axes retain upstream defect information that sink-only labeling omits, distinguishing vulnerabilities that share a consequence.The specification treats cause and consequence as separate elements, and BF records both the first and final weakness.Source: §6.1.2 · p. 18, §6.1.3 · p. 19

6.1.4 The cause-operation-consequence chain addresses the absence of a causal chain (F4).

  • BF links each weakness’s consequence to the next weakness’s cause, representing multi-stage vulnerabilities as causal chains rather than single terminal labels.The specification provides chain representations for Heartbleed and the five-stage BadAlloc pattern; reconstructing a CVE’s full chain remains an inference task.Source: §6.1.4 · p. 19

7 Case Studies

Four case studies contrast contested or changed CWE labels with BF specifications that preserve weakness stages and chains. In the three cases detailed here, CWE labels can describe different stages, while BF records linked causes, operations, and consequences.

7 Case Studies

  • Across the four stratified cases, CWE labels moved or were contested, while BF classifications remained unchanged under reassignment.The cases were selected from a 16-CVE pool by failure density.Source: §7 · p. 20

7.1 CVE-2021-3156 (Sudo Baron Samedit), BF_INP →BF_MEM

  • As analysis deepened, the label moved from root-cause off-by-one to heap sink; both remain accurate chain stages, while BF classifies cause Erroneous Code, operation Write, consequence Buffer Overflow, attribute Heap.The upstream stages map to DVL and MAD before the MUS Write sink; two annotators assigned the four-axis label identically.Source: §7.1 · pp. 21–22

7.2 CVE-2015-0235 (GHOST), BF_MEM

  • For GHOST, BF links a Wrong Size/Allocate cause to Insufficient Size and then to a Write/Buffer Overflow sink, retaining the allocation defect upstream.The CWE label shifted from CWE-119 to CWE-787, with CWE-122 as a descendant; the assigned CWEs do not express the undersized allocation cause.Source: §7.2 · pp. 22–23

7.3 CVE-2021-21834 (GPAC), BF_DAT →BF_MEM (BadAlloc pattern)

  • For GPAC’s BadAlloc pattern, BF specifies five linked weaknesses from verification through write, whereas CWE-119, CWE-190, and CWE-787 describe different stages.The chain proceeds through DVR, TCM, MMN, MAD, and MUS.Source: §7.3 · p. 24

CAUSE OPERATION CONSEQUENCE

The BadAlloc and Heartbleed examples use BF chains to represent vulnerabilities as linked weaknesses rather than single CWE labels. Heartbleed also demonstrates how BF can represent converging chains.

CAUSE OPERATION CONSEQUENCE

  • The BadAlloc pattern for GPAC comprises five linked weaknesses, from an inconsistent size value through wraparound, undersized allocation, pointer repositioning, and buffer overflow.The stages are DVR, TCM, MMN, MAD, and MUS; the case also has an ACE/RCE failure stage.Source: CAUSE OPERATION CONSEQUENCE · pp. 24–25
  • BF’s explicit cause axis records the initial missing or erroneous code, and its cause-consequence links compose the five weaknesses into one representation.The links carry Inconsistent Value to Wrong Argument, Wrap Around to Wrong Size, Insufficient Size to Reposition fault, and Overbound Pointer to Write fault.Source: CAUSE OPERATION CONSEQUENCE · p. 25

7.4 CVE-2014-0160 (Heartbleed), BF_INP →BF_MEM, with convergence

  • Heartbleed’s BF specification represents a three-weakness main chain plus a converging MUS Clear weakness, with either missing Verify or missing Clear implicated in the IEX failure.The chain records missing input verification, pointer repositioning, and buffer over-read; CWE-125 alone cannot represent both chains.Source: §7.4 · pp. 25–26

8 Empirical Evaluation

The evaluation measures manual inter-rater reproducibility and automated chain determinism, not human-validated accuracy. Agreement is strongest on cause and operation, while automated repeatability varies by deployment and does not ensure agreement with published chains.

8.1 Study 1: Inter-Rater Reliability of Manual Mapping

  • Cause and operation had almost-perfect agreement (κ=0.876 and κ=0.900), consequence substantial agreement (κ=0.626), and attribute fair agreement (κ=0.235).The pooled statistic across 52 cells was κ=0.732.Source: §8.1 · p. 27

8.2 Study 2: Automated Determinism of CVE-to-BF Mapping

  • The automated study used evidence-based, stepwise chain generation over repeated rounds, with two deployments evaluated at their own context ceilings.Evidence included NVD advisory text and fix-commit material; the evaluation sampled CVEs and compared the two deployments on a balanced window.Source: §8.2 · p. 27, Algorithm 1 : Backward BF Chain… · pp. 28–29

8.3 The Formal Derivation Procedure

  • The procedure derives chains backward from failure, assigning one BF weakness per link using closed vocabulary tables and a mix of deterministic lookups and semantic selections.The class assignment is deterministic; operation, cause, and attribute selections use semantic inference.Source: §8.3 · p. 27

Algorithm 1 : Backward BF Chain Derivation with Neuro-Symbolic Verification

  • Algorithm 1 stops when a cause resolves to a bug, then verifies vocabulary membership, chain structure, operation-class consistency, and link continuity before release.Accepted chains carry an audit trail of queries, results, and evidence anchors.Source: Algorithm 1 : Backward BF Chain… · p. 28

8.4 Study 2 Results

  • Across full coverage, Gemma reproduced class-chains for 312 of 352 CVEs versus 282 whole chains; GPT-OSS reproduced 19 class-chains versus 4 whole chains.Gemma’s fully deterministic share was higher for high-confidence than medium-confidence evidence (132 of 145, mean D=0.938 versus 132 of 185, mean D=0.793).Source: §8.4 · p. 31
  • On 85 CVEs, Gemma 4 31B achieved D=0.890 and S=0.968, while GPT-OSS 120B achieved D=0.094 and S=0.577.GPT-OSS disagreement was concentrated on whole-chain, sink, and failure axes rather than the root.Source: §8.4 · p. 31, Figure 16 · p. 32
  • Reliable outputs most often reproduced class-chains that differed from the published verified chains, so repeatability did not establish fidelity to those chains.Derived and verified chains concentrated on short two- and three-class paths, but their modal renderings differed.Source: §8.4 · pp. 31–33

8.5 Interpretation and Scope

  • Determinism depended on deployment and baseline configuration; across both studies, the most reproducible axes were those most tightly constrained by the specification.The automated determinism measure concerns repeatability under the tested configurations and evidence representation, not accuracy against expert-adjudicated chains.Source: §8.5 · p. 33

9 Scope and Boundary Conditions

The comparison is bounded by BF’s current taxonomy and the specification snapshot used in the study. CVEs rooted outside the covered class types are outside the reported comparison.

9 Scope and Boundary Conditions

  • BF’s July 2024 taxonomy covers INP, MEM, DAT, and FLR; CWE-covered root causes outside those class types are outside the comparison.BF covers a defined subset of vulnerability classes covered by CWE.Source: §9 · p. 34
  • The reported comparison does not address CVEs rooted outside BF’s four current class types; extension to forthcoming NIST specifications is future work.Source: §9 · p. 34
  • The study uses SP 800-231 as its specification snapshot, while a proposed classifier would need to consume the independently maintained live BF taxonomy.Source: §9 · p. 36

10 Conclusion

The paper evaluates NIST SP 800-231 as a target for automated vulnerability classification and concludes that BF is more structured and automation-suitable than CWE. Its claims remain bounded by the limited case studies and study design.

10 Conclusion

  • The authors position BF as a promising, more automation-suitable target, while calling for better per-axis validation, annotation consistency, and alignment with verified chains.Source: §10 · p. 36

B.1 Construct Validity

  • Four case studies cannot establish that the documented CWE failure patterns are systemic across the wider CVE corpus.Reassignment of a primary CWE is indicative, not conclusive, because it may reflect deeper root-cause analysis.Source: §B.1 · p. 43

B.2 Internal Validity

  • The blind two-annotator design bounds annotator-bias concerns, but a family-level encoding mismatch drove the low attribute agreement and limits determinism claims to cause and operation.Source: §B.2 · p. 43

B.3 Limitations of the Bugs Framework as an Automation Target

The authors identify five specification gaps that constrain CVE-to-BF automation, including under-specified attribute choices and ambiguity among legal chain renderings.

B.3 Limitations of the Bugs Framework as an Automation Target

  • Attribute values are enumerated, but selection guidance is limited; automated cross-round variation concentrates in attribute assignments while class-chains hold.Source: §B.3 · p. 43
  • Unbounded chain length and incompletely enumerated causation rules permit multiple legal renderings; strict value identity holds for only 21 of 38 adjacent links in the verified corpus.Source: §B.3 · p. 43
  • A one-weakness chain can be legal BF even when verified chains use several phases, and the specification does not rank conforming chains of different depths.Source: §B.3 · p. 43
  • Consequence sets are published for only two of nine classes, and cross-class operation flow is incomplete, leaving not-listed distinct from not-allowed unclear.Source: §B.3 · p. 43
  • Available artifacts bound derivable chain depth independently of model capability, so evidence richness constrains how far a chain can be inferred.Source: §B.3 · p. 43

C Database Search Strategies

The search protocol was designed to support independent regeneration and checked retrieval against known-relevant papers. It yielded all included papers and anchor papers without supplementary searching.

C Database Search Strategies

  • Four databases used adapted field-tagged Boolean queries, while other platforms used query approaches suited to their search interfaces.Google Scholar used twelve short queries; arXiv, IEEE Xplore, ACM, and Springer Link each had platform-specific search settings.Source: C Database Search Strategies · p. 44, §C.3 · p. 44, Table 11 · p. 46
  • Searches applied a January 2018–April 2026 publication window and restricted content to journal and conference papers.The four field-tagged databases shared a common keyword concept structure adapted to each database’s syntax.Source: C Database Search Strategies · p. 44

C.1 Anchor Papers: Known-Relevant Validation Set

  • Preselected anchor papers formed a known-relevant validation set for checking keyword-search recall, even when some were excluded from the final corpus.VWC-MAP and ThreatCompass were excluded because CWE served as an intermediate stage toward CAPEC or ATT&CK.Source: §C.1 · p. 44

C.2 Case-Study Candidate Pool

  • The 16-CVE candidate pool contained four candidates per BF bug-type family, and case studies maximized exposed structural CWE failures within each family.Anchor status broke ties.Source: §C.2 · p. 44, Table 9 · p. 45

C.3 Identification Yield, Recall, and Query Protocols

  • 660 distinct records yielded all 20 included papers and every anchor, with no supplementary search method required.Table 10 reports per-database candidate records before deduplication.Source: §C.3 · p. 44, Table 10 · p. 46
Loading 2609.16433v1…