Source-linked AI summary
CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software
Guru Prasad Bhandari, Amara Naseer, Leon Moonen
TL;DR
Security vulnerability research lacks comprehensive datasets of real-world vulnerable code and fixes suitable for data-driven discovery and repair studies. CVEfixes automatically mines NVD records and linked open-source repositories, organizes the results in a relational database, and releases an enriched multi-level dataset. Its initial release covers 5365 CVEs across 1754 projects and 5495 fixing commits, supporting several software-security research tasks.
Problem
Data-driven vulnerability research needs comprehensive datasets containing real-life vulnerable code, fixes, multiple granularities, diverse languages, and labeled samples.
Method
CVEfixes automatically collects NVD CVE records and associated open-source repositories, extracts vulnerable and patched code, and organizes enriched information in a relational database.
Results
5365 CVE records across 1754 open-source projects and 5495 vulnerability-fixing commits comprise the initial CVEfixes release.
Takeaways & Limitations
CVEfixes supports vulnerability detection and classification, severity prediction, vulnerability-related code-change analysis, and automated vulnerability repair.
Takeaways & Limitations
The collection tool gathers all fixes from scratch and may omit information when NVD-referenced repositories are unavailable.
Abstract
from arXiv · showhide
Data-driven research on the automated discovery and repair of security vulnerabilities in source code requires comprehensive datasets of real-life vulnerable code and their fixes. To assist in such research, we propose a method to automatically collect and curate a comprehensive vulnerability dataset from Common Vulnerabilities and Exposures (CVE) records in the public National Vulnerability Database (NVD). We implement our approach in a fully automated dataset collection tool and share an initial release of the resulting vulnerability dataset named CVEfixes. The CVEfixes collection tool automatically fetches all available CVE records from the NVD, gathers the vulnerable code and corresponding fixes from associated open-source repositories, and organizes the collected information in a relational database. Moreover, the dataset is enriched with meta-data such as programming language, and detailed code and security metrics at five levels of abstraction. The collection can easily be repeated to keep up-to-date with newly discovered or patched vulnerabilities. The initial release of CVEfixes spans all published CVEs up to 9 June 2021, covering 5365 CVE records for 1754 open-source projects that were addressed in a total of 5495 vulnerability fixing commits. CVEfixes supports various types of data-driven software security research, such as vulnerability prediction, vulnerability classification, vulnerability severity prediction, analysis of vulnerability-related code changes, and automated vulnerability repair.
1 INTRODUCTION
CVEfixes addresses the need for comprehensive, real-world vulnerability datasets by automatically collecting vulnerable code and corresponding fixes from NVD-linked open-source projects. The initial release provides a publicly shared, multi-level dataset with associated metadata for data-driven security research.
- Data-driven vulnerability research requires large, diverse datasets containing labeled vulnerable and non-vulnerable code, fixes, multiple granularities, and widely used programming languages.
- CVEfixes automatically mines NVD CVE records and associated repositories to collect real-world vulnerable and patched code.
- CVEfixes supports vulnerability identification, CWE classification, severity prediction, and automated vulnerability repair research.
- The dataset is organized at multiple levels of granularity and enriched with programming-language metadata and code-related metrics at five abstraction levels.
- 5365 CVE records from 1754 open-source projects and 5495 vulnerability-fixing commits comprise the initial publicly shared CVEfixes release.
2 RELATED WORK
Existing vulnerability datasets differ in scope, granularity, labeling, and reuse challenges. CVEfixes addresses these gaps by combining broad CVE coverage with source-level vulnerable and fixed code and repeatable collection.
- Many comparison datasets are limited by narrow project or language coverage, class imbalance, incomplete functions, or restricted labels.
- Process-oriented bug-fixing data can provide issue-management information without code-level detail, while repository-mining tools may stop at commit-level data.
- CVEfixes covers vulnerability information at multiple granularities, up to source code before and after fixing.
- CVEfixes differs from these resources by linking vulnerability records, fixes, and detailed code information in a broader, reusable collection.
- Its collection timeframe includes all CVE records available on 9 June 2021 and can be extended through future automated updates.
3 DATASET CONSTRUCTION
The dataset construction workflow collects NVD CVEs and CWE information, retrieves associated repositories and fixing commits, extracts vulnerable and fixed code, computes metadata, and stores the results relationally. The automated tool supports reproducible updates and multiple ways to query or rebuild the collection, although a full from-scratch run can take up to 15 hours.
- Dataset construction: The workflow starts with NVD JSON vulnerability feeds, collects CWE details, and cross-references CWE types with CVE records.
- Dataset construction: For CVEs with associated fixes, the tool clones project repositories and uses reported commit hashes to gather vulnerable and fixed source code.
- Dataset construction: Extracted code receives code-level metrics and programming-language metadata before the collected information is stored in a relational database.
- Dataset construction: The database represents CVEs, CWEs, repositories, commits, changed files, changed methods, fixes, and CWE classifications through interconnected tables.
- Automation and updates: The collection tool revisits all published CVEs during updates because newly added fixes can affect any existing CVE.
- Reproducibility and use: The distribution supports a compressed SQL dump, from-scratch collection, configuration settings, example queries, and SQLite3 storage.
- Reproducibility and use: A full from-scratch collection can take up to 15 hours, depending on the available internet connection.
4 DATASET EXPLORATION
CVEfixes contains 5365 unique CVEs across 1754 open-source projects, linked to 5495 vulnerability-fixing commits and 180 CWE types. Its exploration shows uneven project and vulnerability distributions, generally high severity scores, and many high-risk maintainability changes in fixes.
- Dataset overview: 5365 unique CVEs span 1754 open-source projects and 5495 vulnerability-fixing commits, classified into 180 CWE types.Some commits address multiple CVEs, so the commit count is lower than the CVE count.
- Project distribution: There is no direct correlation between the numbers of files or methods changed and the numbers of CVEs or fixing commits.The CVE and commit counts do show the expected relation, with minor differences because some commits cover multiple CVEs.
- Project distribution: The top-ten project lists cover 31.84% of CVEs, 31.61% of fixing commits, 42.43% of changed files, and 57.84% of changed methods.The different shares show that project rankings vary by whether vulnerabilities, commits, files, or methods are counted.
- Vulnerability types: CWE-79 is the most common vulnerability type, with 635 CVEs and 670 fixing commits changing 3226 files.The top ten CWE types cover approximately 55.58% of all CVEs.
- Severity and maintainability: Most vulnerabilities have high severity scores, while exploitability and impact tend toward the lower sides of their respective ranges.Figure 3 aggregates CVSSv2, CVSSv3, exploitability, and impact scores as project averages over a 0–10 range.
- Severity and maintainability: DMM metrics are concentrated near 0 for unit size, complexity, and interfacing, while the overall DMM score has median around 0.25 and third quartile around 0.6.Scores near 0 indicate high-risk changes with a detrimental effect on maintainability; the overall score aggregates the individual metrics.
5 DISCUSSION
CVEfixes supports multiple data-driven software-security applications by linking vulnerability records to code-level fixes, while its collection process and data coverage retain important limitations and future-extension opportunities.
- Applications of CVEfixes: CVEfixes links CVE-level information to code changes, enabling vulnerability prediction, classification, severity prediction, and automated security repair.Its multi-level data supports extracting code features, metrics, and textual descriptions for machine-learning models.
- Applications of CVEfixes: The dataset can support repair research by isolating fixes with only single or a few modified lines across programming languages and vulnerability types.These cases can focus training on patches with fewer modified tokens, which the paper connects to improving patch quality.
- Limitations and Future Extensions: The current tool collects all available fixes from scratch, and incremental updating could reduce the time required to refresh the database.Removing fixes that lose CVE references would require additional checks because other CVEs may reference the same fixes.
- Limitations and Future Extensions: Unavailable repositories force CVEfixes to remove their references, so the dataset presents only a subset of information available in the NVD.Repositories may have been removed, renamed, or moved between hosting forges.
- Limitations and Future Extensions: A referenced fixing commit may leave part of a vulnerability unresolved or include unrelated changes, motivating analysis of consecutive patches.Later commits may complete the fix, so selecting and disentangling the relevant code remains a future improvement.
- Limitations and Future Extensions: Mining additional issue-tracking and version-control systems could expand the collection to more projects, while license metadata could help users filter fixes for training data.The proposed extensions include Bugzilla, Mercurial, Subversion, and license information for included fixes.
6 CONCLUDING REMARKS
CVEfixes is a fully automated, relationally organized dataset of real-world vulnerable and patched code linked to CVE records and enriched with security and code metrics. Its initial release covers thousands of fixes across thousands of open-source projects, supporting vulnerability-related investigations from CVEs to code changes.
- CONCLUDING REMARKS: CVEfixes combines a fully automated collection tool with vulnerable and patched code, CVE links, security metrics, and relational-database organization.The dataset includes metrics at five abstraction levels and information fetched from the NVD and version-control systems.
- CONCLUDING REMARKS: 5495 vulnerability fixing commits from 1754 open-source projects form the initial multi-level release.The dataset supports investigations connecting published CVEs to code-level vulnerabilities and proposed fixes.