Source-linked AI summary
When Cyber Scoring Systems Diverge: An Empirical Comparison
Kirsi Hellsten, Joni Herttuainen, Ambrose Kam, Arlanda Johnson, David Welsh, Kimmo Kaski
TL;DR
Prior comparisons of vulnerability scoring systems have mostly used aggregate agreement measures in IT databases, leaving their operational consequences in OT networks less examined. This paper compares four systems on a reconstructed Ukraine Power Grid topology and finds structured differences in the signals they capture. It concludes that combining multiple signals may provide a more informative basis for operational remediation decisions, while the proposed formulations remain heuristic.
Problem
Method
The study compares CVSS, EPSS, SSVC, and IronMiner using a reconstruction of the 2015 Ukraine Power Grid OT network and presents two hybrid scoring formulations.
Source: §1 · p. 2Results
Scoring systems capture distinct signals, and their differences are structured: CVSS versions correlate strongly, EPSS365 and IronMiner correlate strongly, while their correlations with CVSS are generally moderate and SSVC’s are comparatively weak.
Source: §5 · pp. 15–17Takeaways & Limitations
Combining severity, exploitation likelihood, and operational decision information may provide a more informative basis for remediation decisions than any individual scoring system.
Source: §5 · p. 16The proposed score formulations rely on heuristic weighting, mapping, and interaction assumptions and are not formally validated measures of risk.
Source: Methodologies · p. 13, §5 · p. 16
Abstract
from arXiv · showhide
Vulnerability scoring systems underpin cyber patch prioritization and risk management, but their comparative behavior is almost always assessed in the abstract, through correlation studies in IT vulnerability databases, rather than by the operational consequences they produce when embedded in a system-level risk model. Here we present an empirical comparison of four vulnerability scoring systems, namely CVSS (Common Vulnerability Scoring System), EPSS (Exploit Prediction Scoring System), SSVC (Stakeholder-Specific-Vulnerability Categorization), and IronMiner (operationally calibrated proprietary scoring system). As a substrate for comparison, we use a reconstruction of the 2015 Ukraine Power Grid operational-technology (OT) network that provides a documented incident topology. The results show a high degree of disagreement between the scoring systems. This suggests that the choice of the scoring system could significantly influence mitigation strategies and vulnerability prioritization, implying that a composite or hybrid scoring approach could offer a more suitable solution.
1 Introduction
The introduction frames vulnerability scores as tools for choosing what to patch, while noting that existing comparisons rarely test their operational consequences in OT networks. The study compares four scoring systems on a reconstructed Ukraine Power Grid topology and examines whether their differences affect network-wide rankings and defensive prioritization.
1 Introduction
- Prior comparisons rarely establish what scoring-system disagreement means for attack propagation and defensive prioritization in OT networks.The introduction highlights OT topology and lateral movement through bridging nodes as factors that can shape attack impact.Source: §1 · p. 2
- The study compares CVSS, EPSS, SSVC, and IronMiner using a reconstruction of the 2015 Ukraine Power Grid OT network.It asks whether network-wide risk rankings diverge, whether divergence is systematic or random, and whether it changes defensive prioritization.Source: §1 · pp. 1–2
- The paper contributes an empirical comparison on a documented OT incident topology, disagreement analysis across CVE, subscore, and network-placement levels, and two hybrid formulations.The formulations are a weighted composite score and a CVSS-anchored priority score, addressing systems that answer different questions.Source: §1 · p. 2
2 Related Works
Prior work finds disagreement among vulnerability scoring systems, but has largely studied aggregate agreement in IT vulnerability databases rather than operational consequences in specific networks. This study addresses that gap in an OT setting.
2 Related Works
- Tan et al. () found CVSS severity and EPSS exploitation likelihood uncorrelated, with EPSS scores changing dynamically over a 360-day window.They concluded the systems provide complementary signals and neither should be used alone for prioritization.Source: §2 · pp. 2–3
- Only 5 vulnerabilities appeared in all four systems’ top-100 lists in Koscinski et al. (), whose study found weak agreement across measures.The study also reported opposite prioritization tendencies and limited EPSS predictive value against the KEV catalog.Source: §2 · p. 3
- Pinaev () proposed a composite model awaiting large-scale validation, while Wunder et al. () found CVSS ratings varied across and within evaluators.Wunder et al. attributed rating variation largely to ambiguous metric definitions such as Scope.Source: §2 · p. 3
- Prior studies establish scoring-system disagreement but leave its consequences in network attack propagation, especially in OT, unexamined.Their evidence is largely aggregate correlation and agreement statistics from IT-centric vulnerability databases.Source: §2 · p. 3
3 Methods and Materials
The section describes four vulnerability scoring systems and a reconstructed Ukraine-grid scenario, then explains how their outputs are mapped and combined for comparison. It also characterizes the vulnerability records and metrics used in the dataset.
Vulnerability Scoring Systems
- CVSS v2 uses a simple base score but has documented limitations, while v3.0 adds attack and impact distinctions and v3.1 clarifies metric definitions without changing formulas.The section notes that CVSS v2 is less precise for modern attack scenarios and has network-centric bias; v3.1 was intended to reduce inconsistent scoring.Source: Vulnerability Scoring Systems · pp. 4–6
- IronMiner derives daily CVE threat scores from threat-intelligence indicators, EPSS estimates near-term exploitation probability, and SSVC maps stakeholder-specific decision factors to actions.IronMiner's GLM predicts KEV appearance; SSVC decision factors and outputs vary with stakeholder role.Source: Vulnerability Scoring Systems · pp. 6–8
Sample Scenario: 2015 Ukraine Power Grid
- The scenario reconstruction uses the reported 2015 outage sequence and a simplified OT topology, contrasting technology available in 2015 with a contemporary configuration.The event sequence runs from phishing and reconnaissance through SCADA HMI access, breaker manipulation, disruption of operator response, and KillDisk deployment.Source: Sample Scenario: 2015 Ukraine Power Grid · pp. 9–10, Figure 1 · p. 10
Methodologies
- SSVC decisions are numerically mapped for comparison, and the study derives decision-tree inputs from KEV status, CVSS metrics, and attack-graph node placement.The authors characterize the SSVC-to-score mapping as heuristic because CVSS severity, EPSS exploitation likelihood, and SSVC operational decisions measure different aspects.Source: Methodologies · pp. 10–11, Table 3 · p. 11
- The study describes both a weighted composite score and a CVSS-anchored priority score, but their mappings, weights, and multiplicative assumptions remain subjective or heuristic.The weighted score combines CVSS, EPSS, and SSVC; the alternative adjusts CVSS using SSVC and EPSS-derived factors and is not presented as a formally validated risk measure.Source: Methodologies · pp. 11–13
Dataset Description
- The dataset contains 659 vulnerability records from 12 systems and 27 products, representing 592 unique CVEs and including CVSS, EPSS/EPSS365, and SSVC measures.SSVC recommendations comprise 621 Track, 27 Attend, 9 Track*, and 2 Act records.Source: Dataset Description · p. 13
4 Results
The scoring frameworks yield non-interchangeable vulnerability-risk representations, with differences between CVSS versions and across exploitation-oriented and decision-oriented measures. Correlations also show structured overlap rather than equivalent rankings.
4 Results
- CVSSv2 and CVSSv3 mean exploitability scores were 8.26 and 2.89, respectively, with a 5.37-point mean absolute difference.The mean absolute difference was 5.37 points (SD = 1.42); mean impact decreased from 6.03 to 4.58, with a 1.92-point average absolute difference (SD = 1.60), while the overall Base Score mean absolute difference was 1.52 (SD = 1.19), maximum 5.70.Source: §4 · pp. 13–14, Table 4 · p. 14
- Extending EPSS to a 365-day horizon increased the median from 0.03 to 0.35 and the mean from 0.17 to 0.47, indicating strong dependence on forecasting horizon.The 75th percentile also rose from 0.17 to 0.90.Source: §4 · p. 14
- IronMiner and CVSSv3 share a mean score of 7.55, but IronMiner's scores range from 1.80 to 10.00 and differentiate more strongly in this dataset.IronMiner's median is 7.90, and its standard deviation is 2.11; matching means do not imply similar rankings.Source: §4 · pp. 14–15
- SSVC's median and both quartiles are 2.00, with a mean of 2.82 and maximum of 8.00, suggesting less differentiation here than continuous scoring systems.At least 75% of observations receive the same numerical value of 2.Source: §4 · p. 15
- The scoring frameworks do not provide interchangeable representations of vulnerability risk, with EPSS, IronMiner, and SSVC capturing dimensions distinct from CVSS.The particularly large difference between CVSSv2 and CVSSv3 exploitability scores also cautions against comparing component scores across versions.Source: §4 · p. 15
- CVSSv2 and CVSSv3 correlate strongly for corresponding impact (rs = 0.82) and exploitability (rs = 0.86); EPSS365 and IronMiner also correlate strongly (rs = 0.77).SSVC correlations with other measures are generally weak, while EPSS365 and IronMiner have only moderate associations with CVSS.Source: §4 · p. 15
5 Conclusions
The study finds that vulnerability scoring systems capture overlapping but non-equivalent risk dimensions, with structured correlation differences. It supports multi-signal assessment, while noting that proposed hybrid formulations remain heuristic and unvalidated.
5 Conclusions
- CVSS emphasizes technical severity and impact, EPSS and IronMiner capture exploitation likelihood, and SSVC provides a decision-oriented assessment incorporating operational context.The paper examines these systems using a reconstructed topology of the 2015 Ukraine Power Grid cyber incident.Source: §5 · pp. 15–16
- Scoring systems disagree in structured ways: CVSS versions correlate strongly, EPSS365 and IronMiner correlate strongly, and SSVC is comparatively weakly correlated with both groups.The systems therefore capture overlapping but non-equivalent dimensions of vulnerability risk.Source: §5 · p. 16
- The study supports multi-signal vulnerability assessment in operational environments and presents weighted-composite and CVSS-anchored priority formulations as possible integration mechanisms.Their weighting, mapping, and interaction assumptions remain heuristic and are not formally validated as risk measures.Source: §5 · pp. 16–17
A Summary of differences between CVSS 2.0 and 3.x
Table 5 compares CVSS 2.0 and CVSS 3.x across key aspects.
A Summary of differences between CVSS 2.0 and 3.x
- Table 5 compares CVSS 2.0 and CVSS 3.x across key aspects.Source: Table 5 · p. 18
B SSVC decision point derivation logic
Table 6 presents SSVC decision point derivation logic.
B SSVC decision point derivation logic
- Table 6 presents SSVC decision point derivation logic.Source: Table 6 · p. 19
C Statistics of all the vulnerabilities
Table 7 reports statistics for vulnerabilities with vulnerability scoring found.
C Statistics of all the vulnerabilities
- Table 7 reports statistics for vulnerabilities with vulnerability scoring found.Source: Table 7 · p. 20