Source-linked AI summary
A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software
Serena E. Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, Cédric Dangremont
TL;DR
Software vulnerability research and management need reliable code-level evidence, but existing vulnerability information can be incomplete or insufficient for identifying fixes. The paper manually curates an open-source dataset from NVD and project-specific sources, mapping vulnerabilities to fixing commits in industrially used Java projects. The resulting dataset covers 624 vulnerabilities, 205 projects, and 1282 fixing commits, and has supported classifiers for identifying security-relevant commits.
Problem
Reliable code-level data is needed to advance vulnerability identification, impact analysis, mitigation, and secure software development.
Method
The authors manually collect and curate vulnerability and fix data from the NVD and continuously monitored project-specific Web resources.
Results
The dataset maps 624 publicly disclosed vulnerabilities across 205 open-source Java projects to 1282 fixing commits and has been used to train classifiers identifying security-relevant commits.
Takeaways & Limitations
Open release of the dataset and supporting code supports future research using industrially relevant data and shared maintenance by communities, academia, and industry.
Abstract
from arXiv · showhide
Advancing our understanding of software vulnerabilities, automating their identification, the analysis of their impact, and ultimately their mitigation is necessary to enable the development of software that is more secure. While operating a vulnerability assessment tool that we developed and that is currently used by hundreds of development units at SAP, we manually collected and curated a dataset of vulnerabilities of open-source software and the commits fixing them. The data was obtained both from the National Vulnerability Database (NVD) and from project-specific Web resources that we monitor on a continuous basis. From that data, we extracted a dataset that maps 624 publicly disclosed vulnerabilities affecting 205 distinct open-source Java projects, used in SAP products or internal tools, onto the 1282 commits that fix them. Out of 624 vulnerabilities, 29 do not have a CVE identifier at all and 46, which do have a CVE identifier assigned by a numbering authority, are not available in the NVD yet. The dataset is released under an open-source license, together with supporting scripts that allow researchers to automatically retrieve the actual content of the commits from the corresponding repositories and to augment the attributes available for each instance. Also, these scripts allow to complement the dataset with additional instances that are not security fixes (which is useful, for example, in machine learning applications). Our dataset has been successfully used to train classifiers that could automatically identify security-relevant commits in code repositories. The release of this dataset and the supporting code as open-source will allow future research to be based on data of industrial relevance; also, it represents a concrete step towards making the maintenance of this dataset a shared effort involving open-source communities, academia, and the industry.
[PRE-PRINT]
The paper presents a manually curated dataset of open-source vulnerability fixes collected from the NVD and project-specific resources. It releases the dataset and supporting scripts for research and automated identification of security-relevant commits.
- The data was manually collected from the NVD and continuously monitored project-specific Web resources.
- 624 publicly disclosed vulnerabilities across 205 open-source Java projects are mapped to 1282 fixing commits.
- 29 vulnerabilities lack CVE identifiers, while 46 CVE-assigned vulnerabilities were not yet available in the NVD.
- Supporting scripts retrieve commit contents, augment instance attributes, and add non-security-fix instances for applications such as machine learning.
- The dataset has been used to train classifiers that identify security-relevant commits in code repositories.
I. INTRODUCTION
The paper addresses the difficulty of maintaining a secure open-source software supply chain by providing code-level, industrially relevant vulnerability-fix data. Its manually curated dataset differs from automated or restricted resources through quality control, broader project-specific coverage, and open availability.
- Effective open-source vulnerability management must analyze vulnerabilities and fixes at code level rather than rely purely on often incomplete or inaccurate metadata.
- The dataset was created to simplify the expensive manual identification of fixes for newly disclosed vulnerabilities and has trained automated commit classifiers.
- 624 vulnerabilities in 205 industrially used Java projects are mapped to 1282 fixing commits, with four years of manual curation.
- Manual curation improves entry quality by addressing NVD inconsistencies and including commits absent from NVD records.
- The project selection targets open-source components used in enterprise products or internal tools, giving the dataset practical industrial relevance.
- Unlike some prior datasets, this dataset contains vulnerability-fixing commits rather than commits that introduce vulnerabilities.
- The dataset is openly licensed and intended to support collaborative maintenance involving industry, academia, and the open-source community.
II. DATASET CONSTRUCTION
The dataset construction builds on SAP Security Research’s code-level vulnerability analysis and an operational process for maintaining current vulnerability knowledge. Researchers continuously monitor vulnerability sources, manually review disclosures, and locate corresponding fix commits.
- SAP’s Vulas tool implements an approach for detecting, assessing, and mitigating open-source vulnerabilities and has been used on hundreds of Java and Python applications.
- A comprehensive, accurate, and timely vulnerability knowledge base is treated as a prerequisite for effective tool operation.
- The construction process continuously monitors the NVD and more than 50 project-specific websites for new disclosures.
- Each disclosure is manually reviewed, and corresponding fix commits are sought in the affected component’s repository.
- The resulting database contains 1282 commits from 205 open-source Java projects, corresponding to fixes for 624 publicly known vulnerabilities.
III. DATASET DESCRIPTION
The dataset represents vulnerabilities and their fixing commits as curated 4-tuples, with supporting code for extending and analyzing the data. It covers 624 vulnerabilities across 205 projects and reveals patterns in fix effort, repository concentration, and disclosure timing.
- Each dataset entry is a 4-tuple comprising a vulnerability identifier, repository URL, commit identifier, and class.
- Supporting scripts can obtain negative instances, fetch repository data, and extend or analyze the CSV dataset.The data and scripts are released under the Apache 2.0 license, and example notebooks and Python scripts support manipulation and analysis.
- The released dataset contains 624 vulnerabilities across 205 projects, including 1,282 unique fixing commits and vulnerabilities absent from the NVD.It includes 29 vulnerabilities without CVE identifiers and 46 with CVE identifiers not yet published in the NVD.
- 364 vulnerabilities are fixed in one commit, while 7 require more than ten commits, with one case reaching twenty-three commits.Multiple commits may represent staged fixes or backports to other branches.
- The dataset is biased toward recent vulnerabilities because disclosures increase over time, older information is often unavailable, and manual curation prioritizes recent cases.
- 178 of 205 repositories contain only one vulnerability, showing that vulnerabilities are concentrated unevenly across repositories.
IV. APPLICATIONS
The dataset supports applications ranging from studying vulnerability-fix release delays to training classifiers that identify security-relevant commits. Its open release and shared benchmark aim to support reproducible research on industrially relevant OSS.
- Release-delay analysis: 817 fix commits were released in under 100 days, including 181 released the same day, while many others experienced substantially longer delays.These delays matter because fixes may be public before clients can update to a non-vulnerable release.
- Release-delay analysis: 167 fix commits were not reachable from any release tag, suggesting that some fixes remained in repositories without entering a release.The observation follows the common practice of tagging releases in open-source projects.
- Automated classification: The dataset was used to train patch- and message-based classifiers that flag commits as security-relevant using high-precision models and voting.The classifiers operate without vulnerability-advisory information and combine independent patch and log-message analyses.
- Automated classification: Duplicate fixes across repository branches are removed before classifier training, reducing the data to 862 unique positive instances.The duplicates have different identifiers but identical content because the same change is committed across supported branches.
- Dataset reuse: Supporting scripts add non-security commits and expose repository history and Git features, enabling machine-learning datasets and further vulnerability-fix studies.Researchers can clone referenced repositories, extract commit features, and augment each instance's attributes.
- Dataset reuse: The openly released dataset is intended to provide a freely available, machine-readable benchmark for evaluating claimed classifier improvements.The benchmark covers open-source projects of actual industrial relevance and is meant to encourage further work.
V. CONCLUDING REMARKS
The paper presents a manually curated dataset of vulnerability fixes in industrially relevant Java OSS projects. The authors identify Java coverage and continuous maintenance as boundaries, while pursuing broader language coverage and community-supported updates.
- Contribution: The dataset contains vulnerability fixes from Java OSS projects of industrial relevance, built from experience operating SAP's open-source vulnerability management solution.The conclusion frames the dataset as the paper's central contribution.
- Scope and future work: The released data currently cover only Java projects, although the tool now supports more languages and the authors are working to extend the dataset.Language coverage is therefore an explicit scope boundary of the current release.
- Scope and future work: Long-term usefulness requires continuous updates incorporating newly disclosed vulnerabilities.The authors are working toward community maintenance involving developers, academia, industry, and the open-source community.