PaperlayerView exact revision on arXiv

Source-linked AI summary

My Software has a Vulnerability, should I worry?

Luca Allodi, Fabio Massacci

arXiv:1301.1275v3cs.CR

TL;DR

The paper examines whether CVSS-based rules and vulnerability datasets reliably identify software vulnerabilities that will be exploited in the wild. It compares NVD, EDB, and EKITS with Symantec’s in-the-wild data using case-controlled analysis, finding low risk reduction from CVSS alone and higher reduction when black-market exploit information is included.

  • Problem

    The paper addresses whether CVSS-based rules and public vulnerability datasets reliably measure actual exploitation risk.

  • Method

    The study compares NVD, EDB, and EKITS with Symantec’s in-the-wild dataset and uses case-controlled analysis of CVSS scores and subscores.

  • Results

    CVSS-alone policies yield 3-4% risk reduction, while black-market exploit information with medium-high CVSS can yield 61% risk reduction; no tested metric has high specificity.

  • Takeaways & Limitations

    CVSS alone is not economically effective for patching, while black-market exploit information can provide substantially greater risk reduction.

  • Takeaways & Limitations

    The conclusions cannot be generalized to targeted attacks because such attacks are less likely to appear in Symantec’s general anti-virus data.

Abstract

from arXiv · show

(U.S) Rule-based policies to mitigate software risk suggest to use the CVSS score to measure the individual vulnerability risk and act accordingly: an HIGH CVSS score according to the NVD (National (U.S.) Vulnerability Database) is therefore translated into a "Yes". A key issue is whether such rule is economically sensible, in particular if reported vulnerabilities have been actually exploited in the wild, and whether the risk score do actually match the risk of actual exploitation. We compare the NVD dataset with two additional datasets, the EDB for the white market of vulnerabilities (such as those present in Metasploit), and the EKITS for the exploits traded in the black market. We benchmark them against Symantec's threat explorer dataset (SYM) of actual exploit in the wild. We analyze the whole spectrum of CVSS submetrics and use these characteristics to perform a case-controlled analysis of CVSS scores (similar to those used to link lung cancer and smoking) to test its reliability as a risk factor for actual exploitation. We conclude that (a) fixing just because a high CVSS score in NVD only yields negligible risk reduction, (b) the additional existence of proof of concepts exploits (e.g. in EDB) may yield some additional but not large risk reduction, (c) fixing in response to presence in black markets yields the equivalent risk reduction of wearing safety belt in cars (you might also die but still..). On the negative side, our study shows that as industry we miss a metric with high specificity (ruling out vulns for which we shouldn't worry). In order to address the feedback from BlackHat 2013's audience, the final revision (V3) provides additional data in Appendix A detailing how the control variables in the study affect the results.

I. INTRODUCTION

The paper questions whether public vulnerability datasets and CVSS-based remediation rules meaningfully identify vulnerabilities that are actually exploited in the wild. It compares NVD, EDB, and EKITS against Symantec’s in-the-wild dataset using a case-controlled analysis.

  • The study asks whether public vulnerability datasets measure software security and whether CVSS-based government policies reduce attack risk.
  • NVD records disclosed vulnerabilities, EDB records proof-of-concept exploits, and EKITS records vulnerabilities used in black-market exploit kits.
  • The datasets are benchmarked against vulnerabilities exploited in the wild using Symantec’s Threats and Attack Signatures data.
  • A case-controlled randomized experiment estimates the conditional probability that CVSS scores or subscores are associated with attacks.
  • NVD and EDB are not reliable indicators of in-the-wild exploitation, while CVSS provides significant sensitivity only for vulnerabilities bundled in EKITS.
  • Using CVSS alone yields a 3% expected risk reduction, whereas adding proof-of-concept or black-market exploit information can raise the reduction to 60%.

II. DATASETS

The paper constructs four vulnerability datasets covering disclosed vulnerabilities, proof-of-concept exploits, black-market exploit kits, and exploits observed in the wild. These datasets differ in scope and collection method, and SYM does not capture targeted attacks against specific companies.

  • NVD is the reference database for disclosed vulnerabilities maintained by NIST.The dataset contains 49,599 vulnerabilities.
  • EDB contains information on proof-of-concept exploits and reports 8,122 vulnerabilities with proof-of-concept code.
  • EKITS contains vulnerabilities bundled in exploit kits sold on the black market, covering 114 unique CVEs across more than 90 exploit kits.
  • SYM contains 1,277 CVEs identified in viruses and remote attacks by Symantec’s commercial products.
  • SYM does not represent direct attacks by individual motivated hackers against specific companies.

III. EXPLORATORY ANALYSIS OF DATASETS

The exploratory comparison asks whether vulnerabilities in NVD, EDB, and EKITS are likely to appear among Symantec’s threats. EKITS vulnerabilities show the strongest overlap with observed exploitation, while CVSS categories alone overstate risk in NVD and EDB.

  • Dataset overlap: 75.73% of EKITS vulnerabilities are monitored as actively exploited, whereas this likelihood is much lower for EDB-EKITS and NVD-(EDB+EKITS).Table II excludes overlapping vulnerabilities when computing each dataset’s conditional probability.
  • Dataset overlap: EDB and SYM share 393 vulnerabilities, leaving about 75% of Symantec-measured threats absent from EDB.NVD also contains many vulnerabilities that do not appear in SYM.
  • CVSS distributions: Considering HIGH and MEDIUM CVSS vulnerabilities in EDB produces about 94% false positives because most such vulnerabilities are absent from SYM.The exploratory comparison therefore suggests that high scores overestimate exploitation risk in NVD and EDB.
  • CVSS distributions: 21% of SYM vulnerabilities have CVSS <6 and 52% have CVSS <9, so one in five observed exploits is low-risk and one in two is non-high-risk.These figures are 272 and 665 vulnerabilities out of 1,277, respectively.
  • Interpretation: The observational results require caution because dataset size and factors such as vulnerability age or software type may affect whether SYM records an exploit.The paper addresses these concerns later using sensitivity, specificity, subscore analysis, and controlled sampling.

IV. SENSITIVITY AND SPECIFICITY

The paper evaluates CVSS as a test for whether vulnerabilities are exploited in the wild by measuring sensitivity and specificity across datasets. EKITS has high sensitivity, but all datasets have poor specificity, limiting CVSS’s usefulness for ruling out uninteresting vulnerabilities.

  • Metric definitions: Sensitivity measures CVSS-positive vulnerabilities among SYM exploits, while specificity measures CVSS-negative vulnerabilities absent from SYM.The study treats MEDIUM and HIGH scores as positive tests and LOW scores as negative tests.
  • Results: EKITS predicts 97 of 100 vulnerabilities exploited in the wild as dangerous using HIGH or MEDIUM CVSS scores.The paper identifies EKITS as the best dataset for sensitivity.
  • Evaluation criterion: A good indicator would require both sensitivity and specificity to be high, preferably above 90%.The paper uses this standard to assess the dataset-level results.
  • Results: More than half of vulnerabilities not dangerous would be wrongly tagged HIGH or MEDIUM, indicating poor specificity across all databases.Thus CVSS generates many false positives when used to identify vulnerabilities unlikely to be exploited.

V. THE IMPACT AND EXPLOITABILITY SUBSCORES

The CVSS score combines Impact, which measures potential damage, and Exploitability, which attempts to measure likelihood of exploitation. Impact distributions differ substantially across datasets, and vulnerabilities exploited in the wild are not uniformly high-impact.

  • CVSS combines Impact, measuring potential damage, with Exploitability, intended to measure likelihood of exploitation.
  • 20% of SYM vulnerabilities have LOW Impact, while only 8% of EKITS vulnerabilities do.
  • EDB contains 65% MEDIUM Impact vulnerabilities, with the remaining 35% equally split between HIGH and LOW Impact.

A. Breakdown of the Impact subscore

The Impact and Exploitability subscores behave differently across vulnerability datasets. Impact distinguishes vulnerabilities found in the wild and black markets from broader populations, whereas Exploitability is concentrated in a narrow high range and contributes little variation to total CVSS.

  • Breakdown of the Impact subscore: Availability almost always matches Integrity, and removing it changes the Impact score by less than 1% on average.
  • Breakdown of the Impact subscore: Most NVD vulnerabilities score partial on both Impact submetrics, while almost 70% of EDB vulnerabilities score partial in at least one.
  • Breakdown of the Impact subscore: In SYM and EKITS, most vulnerabilities score Complete on the Confidentiality and Integrity submetrics.
  • Breakdown of the Impact subscore: Almost all vulnerabilities across datasets have Exploitability subscores between 8 and 10, with little qualitative separation between HIGH and MEDIUM.
  • Breakdown of the Impact subscore: Because Exploitability resembles a constant, it has little to no influence on variation in the final CVSS score.
  • Breakdown of the Impact subscore: 87.31% of discovered vulnerabilities are network-based, and only 3% are exclusively locally exploitable.

VI. EXPLOITATION TRADE-OFFS

The study uses case-controlled sampling and bootstrapping to compare CVSS characteristics with actual exploitation in SYM. Exploited vulnerabilities tend to involve greater access complexity and may reflect attacker trade-offs between exploitation effort and impact, but dataset differences and confounding variables constrain interpretation.

  • VI. EXPLOITATION TRADE-OFFS: SYM and EKITS contain more medium-complexity vulnerabilities than EDB: 38.35% and 63.11%, versus 25.49%.
  • VI. EXPLOITATION TRADE-OFFS: 55.34% of EKITS vulnerabilities in the analyzed category support a trade-off between greater exploitation effort and higher impact.
  • VI. EXPLOITATION TRADE-OFFS: Potential confounding includes commercial interest and time-varying differences in whether Symantec reports or detects attacks.
  • VI. EXPLOITATION TRADE-OFFS: Samples were drawn from EKITS, NVD, and EDB by matching confounding characteristics appearing in SYM, with Impact-controlled samples producing the highest statistical significance.
  • VI. EXPLOITATION TRADE-OFFS: The analysis bootstrapped sampling with repetition 400 times for each control and dataset to reduce noise from random sampling.
  • VI. EXPLOITATION TRADE-OFFS: The case-controlled study treats CVSS as the explanatory variable and SYM presence as the response variable.

A. Sensitivity and specificity for case-controlled study

The case-controlled evaluation finds that CVSS sensitivity is relatively high but specificity is poor across datasets, and performance varies with the reference dataset.

  • A. Sensitivity and specificity for case-controlled study: CVSS sensitivity is quite high among all evaluated datasets, but specificity is poor for all datasets.
  • A. Sensitivity and specificity for case-controlled study: The 0% variation in EKITS sensitivity and specificity reflects its small size relative to SYM, causing all EKITS vulnerabilities to be selected during sampling.
  • A. Sensitivity and specificity for case-controlled study: CVSS performance differs according to the reference dataset used.

B. Rule-based policies for risk mitigation with CVSS

The study tests whether CVSS-based rule policies meaningfully reduce exploitation risk, comparing CVSS with evidence from vulnerability datasets and exploits observed in the wild. CVSS alone provides weak discrimination, while black-market exploit evidence is substantially more informative.

  • Implications: CVSS-based exploitation testing lacks enough precision for effective patching policies, and CVSS-only rules may therefore be inadequate security countermeasures.Combining the score with external evidence is more informative than using CVSS alone.
  • Research question: The study asks how much risk reduction CVSS provides when combined with different vulnerability datasets.It distinguishes the score’s ability to identify exploitation from its effectiveness as a patching metric.
  • Results: +61% relative risk reduction is associated with fixing HIGH/MEDIUM vulnerabilities identified in EKITS, compared with the next lower risk level.The comparison uses vulnerabilities remaining after patching as the reference.
  • Results: +10% more exploitation chances occur at the high-risk level than at the low-risk level for EDB, whereas NVD shows only a +3% increase.The evidence from proof-of-concept exploits is weaker than evidence from black-market exploit kits.
  • CVSS characteristics: Exploited-in-the-wild vulnerabilities are usually remotely exploitable, require no multiple authentication, and combine confidentiality and integrity losses.Availability impact is irrelevant for more than 96% of SYM vulnerabilities, while only 3% are assessed as only locally exploitable.
  • CVSS characteristics: ExploitDB proof-of-concept exploits are usually easier to exploit than vulnerabilities actually exploited by attackers, while medium-complexity vulnerabilities tend to matter when impact is high.The study also reports that black-market exploit presence can indicate high risk because EKITS vulnerabilities significantly overlap with those exploited in the wild.

IX. THREATS TO VALIDITY

The paper identifies validity concerns in constructing and interpreting the SYM and EKITS datasets, and limits generalization to targeted attacks. Its related-work comparison also positions the study as extending prior analyses from vulnerability trends and proof-of-concept exploits toward exploitation in the wild.

  • Construct validity: SYM construction required manual checks because its unstructured data made CVE relevance to threats uncertain.A random sample of about 50 entries was manually analyzed, and Symantec informally confirmed the relevance of the CVEs.
  • Construct validity: SYM dataset interpretation is complicated because attack and infection dynamics are not always straightforward and network and host-based threats can overlap.The study nevertheless uses both relevant datasets for a general evaluation of risk, including EKITS because exploit kits use drive-by downloads.
  • Construct validity: EKITS may contain incomplete or incorrect exploited-CVE records because of the opaque nature of black-market tools.The authors mitigate this by cross-referencing security-community knowledge with direct observation of black markets.
  • External validity: The results cannot be generalized to targeted attacks against specific platforms or systems because such attacks are less likely to appear in general antivirus data.The authors regard Symantec’s worldwide dataset as representative of general real-world scenarios, but not of targeted attacks.
  • Relation to prior work: Prior vulnerability life-cycle and discovery studies modeled trends but did not provide insight into exploitation, while Bozorgi et al. found that CVSS exploitability did not match ExploitDB data.This study extends that observation through deeper subscore analysis and examination of actual exploitation.
  • Relation to prior work: Research on vulnerability discovery, underground markets, and exploit kits provides context for the study’s dataset choices and market interpretation.Earlier work examined discovery models and underground-market reputation, while this paper reports that EKITS vulnerabilities are actually exploited in the wild.

XI. CONCLUSION

The study finds that CVSS-based rule policies are not reliable or economically effective for identifying vulnerabilities likely to be exploited in the wild. Black-market exploit-kit evidence provides the strongest risk reduction, while CVSS and proof-of-concept evidence provide little additional protection.

  • The study compares NVD, EDB, EKITS, and SYM to evaluate whether CVSS and vulnerability databases reliably indicate exploitation in the wild.
  • CVSS is not expressive enough to serve as a reliable exploitation test with both high sensitivity and high specificity.
  • 61% risk reduction is associated with patching medium-high-CVSS vulnerabilities listed in black-market exploit kits.
  • 10% risk reduction is associated with patching medium-high-CVSS vulnerabilities having proof-of-concept exploits in EDB under specified exploitability conditions.
  • 3-4% risk reduction is obtained from CVSS-alone rule-based policies, making compliance economically ineffective in terms of actual security.
  • NVD and EDB are not reliable sources for assessing exploits in the wild and can instead indicate the relative position of vendors and security researchers.

APPENDIX

The appendix reports bootstrapped case-controlled analyses across primary and supplementary controls, with medians, confidence intervals, and p-values used to assess sensitivity, specificity, and risk reduction. It also documents sampling constraints and substantial variation in risk-reduction estimates.

  • Statistical analysis: Sensitivity, specificity, and risk reduction use bootstrap medians, with 95% confidence intervals and median p-values reported for statistical significance.
  • Experimental design: 80 experiments combine four primary controls with five supplementary controls, while CIA CVSS assessment remains continuously active.
  • Primary controls: The primary controls censor years, censor software, match years, and match software counterparts in SYM.
  • Supplementary controls: The supplementary controls represent disclosure in NVD, proof-of-concept exploits in EDB, black-market exploit-kit trading in EKITS, and combined exclusion conditions.
  • Limitations: Limited EKITS size can make risk reduction uncomputable, while software matching may introduce statistical noise because shared components can span multiple products.
  • Results: Risk-reduction estimates vary across controls and can be negative for NVD, implying that patching HIGH or MEDIUM CVSS vulnerabilities may sometimes target vulnerabilities less likely to be attacked than LOW-score ones.
Loading 1301.1275v3…