Source-linked AI summary
Exploit Prediction Scoring System (EPSS)
Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, Idris Adjerid
TL;DR
The paper addresses an information-security industry described as still immature despite major investments. It presents EPSS as a threat-scoring system and reports remarkably good predictions, while noting performance decay and missed exploit activity as limitations.
Problem
The paper addresses an information-security industry described as still immature despite major investments.
Method
The paper presents EPSS as a threat-scoring system intended to provide defensible threat information for security risk management and policymaking.
Results
EPSS makes remarkably good predictions despite a complex and evolving vulnerability landscape.
Takeaways & Limitations
EPSS contributes threat information to enterprise security risk management and national cybersecurity policymaking.
Takeaways & Limitations
EPSS may experience performance decay over time and may miss exploit activity because it relies on intrusion detection systems.
Abstract
from arXiv · showhide
Despite the massive investments in information security technologies and research over the past decades, the information security industry is still immature. In particular, the prioritization of remediation efforts within vulnerability management programs predominantly relies on a mixture of subjective expert opinion, severity scores, and incomplete data. Compounding the need for prioritization is the increase in the number of vulnerabilities the average enterprise has to remediate. This paper produces the first open, data-driven framework for assessing vulnerability threat, that is, the probability that a vulnerability will be exploited in the wild within the first twelve months after public disclosure. This scoring system has been designed to be simple enough to be implemented by practitioners without specialized tools or software, yet provides accurate estimates of exploitation. Moreover, the implementation is flexible enough that it can be updated as more, and better, data becomes available. We call this system the Exploit Prediction Scoring System, EPSS.
Introduction
Vulnerability remediation is often prioritized with severity scores that do not measure exploitation threat, leaving organizations without a proper forecast of which vulnerabilities will be used. The paper introduces EPSS, an open and transparent scoring system designed to predict exploitation within twelve months using interpretable, publicly available data.
- CVSS measures vulnerability severity and impact but not the threat or probability that a vulnerability will be exploited.
- Misinterpreting CVSS as cyber security risk has made it a de facto remediation-prioritization standard despite its stated scope.
- The paper identifies forecasting vulnerability threat as a critical gap and develops EPSS to predict exploitation within twelve months of public disclosure.
- EPSS uses logistic regression, publicly available inputs, and formalized coefficients to produce an interpretable score implementable with simple tools.
- The scoring system is intended to help defenders allocate remediation resources and help policymakers communicate vulnerability threat.
- The model selects sixteen variables as the best tradeoff between performance and parsimony, including vulnerability attributes and CVE reference counts.
- Proof-of-concept and weaponized exploits, vendor associations, remote code execution, and CVE reference counts are correlated with exploitation probability.
- EPSS outperforms CVSS on both ROC and precision-recall metrics in the reported comparison.
Calibration and Distribution
EPSS predictions are fairly well calibrated, with predicted probabilities generally tracking observed exploitation rates. Most predictions are low, while exploited and non-exploited vulnerabilities remain separated but not perfectly.
- EPSS predictions generally follow the perfect-calibration line and fall within the 95% confidence interval.The calibration plot compares predicted probabilities with observed exploitation proportions.
- Figure 6 compares calibration, prediction-density distributions, and equivalent-coverage effort between EPSS and CVSS.
- 76% of predictions are 2% or less.The prediction distribution is shown on a log scale to amplify its lower end.
- The estimated probabilities clearly separate observed exploited from non-exploited vulnerabilities, but the separation is not perfect.
Randomized 5-fold Cross-Validation
The paper evaluates model performance with randomized five-fold cross-validation and compares it with CVSS on ROC and AUC curves. It includes this analysis as a comparison because randomized splits can be misleading for time-series data.
- Randomized cross-validation can generate predictions across the entire sample, but may mislead when observations are time-dependent.Historical observations may not strongly predict future outcomes when the underlying system shifts over time.
- The randomized-split analysis explores outcomes when time is not factored into model validation.
- The analysis uses five-fold cross-validation and evaluates both ROC and AUC curves.
- CVSS-based performance is plotted as points on the same scale for comparison.
Focusing on Existing Vulnerabilities
The study focuses on vulnerabilities encountered in corporate environments rather than all published CVEs. Among the study window’s CVEs, 44% were reported as open in a corporate environment, and the PR curve improved more than the ROC curve.
- 44% of CVEs published during the study window were ever reported as open in a corporate environment.The study subset reflects vulnerabilities with scanner signatures and observed scans in the wild.
- Not all software is used by enterprises, and scanner signatures are not available for every vulnerability.
- The selected vulnerabilities have detection signatures from scan vendors and recorded occurrences in real-world scans.This focuses the analysis on vulnerabilities vulnerability-management scanners report to companies.
- The ROC curve shows little improvement, whereas the PR curve shows a clear improvement.
- Scanner and corporate environments may yield more data earlier for popular vendors and technologies.The passage offers this as one possible reason for lift in the middle portion of the curve.
Formalizing and Implementing the EPSS
EPSS computes exploitation probability by converting model-derived log odds into a probability, using encoded vulnerability attributes and a transformed reference count. The BlueKeep example produces a predicted 95.2% probability within twelve months of publication.
- The paper illustrates the scoring procedure by computing an example vulnerability’s estimated exploitation probability.
- Model variables are encoded as binary indicators, except reference count, which is log-transformed after adding one.Adding one accommodates a reference count of zero.
- EPSS estimates exploitation probability by applying a logistic transformation to the model’s cumulative log odds.The log odds are formed from vulnerability observations multiplied by model coefficients.
- For CVE-2019-0708, the computed log odds are 2.984.The example is BlueKeep, a widely discussed Microsoft Remote Desktop remote-code-execution weakness.
- 95.2% is the estimated probability that CVE-2019-0708 will be exploited within twelve months of publication.The calculation applies Equation 3 to the example’s log odds.
Discussion and Limitations
EPSS is designed as a simple, interpretable, open-input model for estimating exploitation threat, but its scope is constrained by data visibility, CVE coverage, and an evolving vulnerability landscape.
- Design goals: EPSS prioritizes simple implementation, interpretability, parsimony, and performance, requiring concessions among these goals.The model is intended to remain practical while balancing competing design objectives.
- Data and scope: Observed exploit data identified through signature-based intrusion detection may miss attacks lacking signatures or occurring outside monitored networks.Unobserved exploits cannot inform the model's predictions.
- Data and scope: The 12-month prediction window requires excluding observations newer than twelve months from data collection.This bounds both the target outcome and the usable observations.
- Data and scope: Closed exploitation data prevents fully open measurement, so EPSS accepts open inputs even though model fitting requires access to closed outcome data.The authors distinguish implementing the model from training or fitting it from entirely open data.
- Evolution: EPSS performance may decay as the vulnerability landscape changes, requiring updates and retraining over time.Future improvement can incorporate additional exploitation sources, vulnerability features, longer analysis periods, and new data types.
- Interpretation: EPSS estimates exploitation threat rather than complete risk because it excludes firm assets, operating environments, compensating controls, and patching costs.CVSS severity and EPSS exploitation probability therefore do not by themselves represent organizational risk.
- Data and scope: The framework omits non-CVE software or hardware flaws and misconfigurations because disparate sources lack a common identification method.Future data sources could enable their incorporation.
Conclusion
The conclusion presents EPSS as an early, data-driven step toward vulnerability prioritization that complements rather than replaces broader risk judgment. Its practical value lies in adding defensible threat information to existing policies while acknowledging that security risk cannot be reduced to one score.
- Contribution: EPSS is an early step in vulnerability prioritization research intended to provide useful, defensible threat information.The authors frame the system as a contribution that can motivate additional research.
- Interpreting scores: Security risk is not reducible to CVSS alone or to the combination of CVSS and EPSS scores.Complete decisions still require environmental, system, and information context.
- Interpreting scores: CVSS and EPSS communicate different vulnerability properties: severity and probability of exploitation, respectively.The conclusion treats these measures as complementary rather than interchangeable.
- Interpreting scores: Substituting EPSS for CVSS would ignore severity characteristics, while multiplying EPSS by CVSS is mathematically unsound because CVSS is ordinal.The authors instead recommend considering the two values together but separately.
- Practical use: High-severity, high-probability vulnerabilities should receive early prioritization, while low-severity, low-probability vulnerabilities could be deprioritized.Intermediate cases require additional consideration of organizational context.
- Practical use: Augmenting existing security policies with EPSS information may increase policy efficiency and support broader policy reevaluation.The authors present integration as more realistic than assuming enterprises can readily replace established practices.
- Future role: EPSS is intended as an evolving practice that can contribute to enterprise security risk management and national cybersecurity policymaking.The authors expect its usefulness to depend on continuing implementation and refinement.