Source-linked AI summary
DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection
Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, David Wagner
TL;DR
DiverseVul addresses the need for larger, more diverse vulnerable-code data and evaluates deep-learning approaches for vulnerability detection. The study finds persistent generalization and label-noise challenges, while larger datasets and LLMs offer promising directions.
Problem
Vulnerability detection remains difficult, and evidence is limited about whether deep learning, data scale, and model architecture improve performance, especially on new projects.
Method
The authors release DiverseVul, curated from security reports and vulnerability-fixing commits, then study 11 architectures across four model families using combined datasets.
Results
On combined prior datasets and DiverseVul, the best LLM achieves F1 score of 47.2 versus 29.8 for ReVeal, while models perform poorly on unseen projects.
Takeaways & Limitations
LLMs and source-code-specific pretraining objectives are promising research directions, while deployment requires stronger generalization to unseen projects.
Takeaways & Limitations
Label noise in DiverseVul and prior datasets may introduce errors into test-set performance measurements.
Abstract
from arXiv · showhide
We propose and release a new vulnerable source code dataset. We curate the dataset by crawling security issue websites, extracting vulnerability-fixing commits and source codes from the corresponding projects. Our new dataset contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable functions extracted from 7,514 commits. Our dataset covers 295 more projects than all previous datasets combined. Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities. We study 11 model architectures belonging to 4 families. Our results show that deep learning is still not ready for vulnerability detection, due to high false positive rate, low F1 score, and difficulty of detecting hard CWEs. In particular, we demonstrate an important generalization challenge for the deployment of deep learning-based models. We show that increasing the volume of training data may not further improve the performance of deep learning models for vulnerability detection, but might be useful to improve the generalization ability to unseen projects. We also identify hopeful future research directions. We demonstrate that large language models (LLMs) are a promising research direction for ML-based vulnerability detection, outperforming Graph Neural Networks (GNNs) with code-structure features in our experiments. Moreover, developing source code specific pre-training objectives is a promising research direction to improve the vulnerability detection performance.
1 INTRODUCTION
The paper releases DiverseVul and uses it with prior datasets to evaluate deep-learning vulnerability detection. Results expose weak practical performance, severe unseen-project generalization challenges, dataset-dependent architecture rankings, and promising roles for LLMs and code-specific pretraining.
- Dataset: DiverseVul contains 18,945 vulnerable and 330,492 non-vulnerable functions from 7,514 commits, spanning 150 CWEs.It is more than twice the size of CVEFixes and covers almost 50% more projects than all previously published datasets combined.
- Evaluation: The study evaluates 11 deep-learning architectures from four families across DiverseVul and previously published vulnerability datasets.The families are GNN, RoBERTa, GPT-2, and T5.
- Model comparisons: 47.2 F1 is achieved by the best LLM versus 29.8 for ReVeal when all prior datasets are combined with DiverseVul.On CVEFixes alone, ReVeal scores 12.8 F1 while LLMs score 8.5–16.3, showing that architecture rankings depend on training-data volume.
- Data scaling: A 5× increase in training data raises the best model’s F1 from 10.5 to 48.9, while gains for the strongest models may stagnate after adding DiverseVul.The authors report improvement for 7 of 11 models, but little or no improvement for the three best-performing models.
- Limitations and directions: The best model reaches 47.2% F1, 43.3% true positive rate, and 3.5% false positive rate, which remains impractical for analysts.The paper also finds DiverseVul vulnerable-function labels 60% accurate, despite persistent errors from multi-function vulnerabilities and non-vulnerable changed functions.
- Generalization: Unseen-project evaluation reduces F1 from 49% on seen projects to 9.4%, revealing a major deployment generalization challenge.The authors suggest models may overfit project-specific patterns or coding idioms, but leave the cause unclear.
2 RELATED WORK
Prior vulnerability datasets use synthetic cases, static-analyzer labels, or vulnerability-fixing commits, with varying coverage and label quality. DiverseVul instead targets security issues in real-world projects, expanding project diversity for deep-learning research.
- Synthetic datasets such as SATE IV Juliet and SARD provide common benchmark cases, while some datasets combine synthetic and real-world vulnerability selections.
- Static-analyzer datasets derive function-level labels from analyzer alerts, but their label quality is uncertain because static-analyzer accuracy tends to be low.
- BigVul, CrossVul, and CVEFixes collect vulnerability-fixing commits from NVD CVE records and cover multiple projects, languages, and CWEs.
- DiverseVul crawls security issues because developer and analyst review supports high-quality labels and because these issues represent vulnerabilities in real-world projects.
- DiverseVul adds vulnerabilities from 295 projects absent from previous real-world datasets, making it the most diverse dataset by project coverage.
3 DATA COLLECTION
DiverseVul is constructed by crawling security issues, linking them to commits and projects, and extracting changed and unchanged C/C++ functions with vulnerability labels. The resulting data is categorized using CVE-linked or manually mapped CWE information.
- The collection process begins with security issue websites, retaining issue titles, bodies, and relevant git commit URLs after narrowing 29 sites to two with the most commits.
- The authors parse commit URLs, clone projects, identify C/C++ files, and extract changed functions together with unchanged functions from those files.
- Before-commit versions of changed functions are labeled vulnerable, after-commit versions nonvulnerable, and unchanged functions non-vulnerable.
- CVE-linked issues obtain CWE information through the NVD API, while developer-annotated categories are manually mapped to the 25 most popular CWEs.
- About 85% of the data maps to 150 CWE categories, with CVE records sometimes mapped to multiple CWEs.
4 EXPERIMENTS
The experiments compare 11 architectures across four model families and multiple vulnerability datasets. Results favor code-specific pretrained LLMs on larger data, while performance remains limited and additional data yields diminishing gains for the strongest models.
- Experimental design: The study evaluates 11 architectures from four families: GNN, RoBERTa, GPT-2, and T5.
- Graph Neural Network: ReVeal represents the GNN family by constructing code-property graphs that combine AST, CFG, DFG, and PDG information before graph embedding and classification.
- RoBERTa family: The RoBERTa family includes RoBERTa, CodeBERT, and GraphCodeBERT, which share 12 Transformer encoder layers and masked-language-modeling pretraining.
- RoBERTa family: CodeBERT pretrains on 2.3M functions from six programming languages using masked language modeling and replaced-token detection with paired descriptions and code.
- Results: On all available data, ReVeal achieves 29.76 F1, whereas LLMs achieve 31.96–47.15 F1; NatGen reaches 47.15 F1 but still has a 3.47% false positive rate.
- Results: Code-specific C/C++ pretraining substantially improves vulnerability detection, with CodeT5 and NatGen among the best-performing models.
4.3 Dataset Volume
Training-data volume improves vulnerability-detection performance when training and test data share a distribution, but gains do not reliably transfer to unseen projects. Larger datasets can substantially change architecture rankings, while generalization remains poor.
- Dataset Volume: Increasing training data from the same distribution produces an upward trend in test F1 score.The experiment fine-tuned 100 CodeT5 Small models across ten runs using different training and validation volumes.
- Results: Models perform very poorly when tested on projects absent from training, creating a major deployment challenge.The unseen-project setup trains on seen projects and evaluates on 95 unseen projects.
- Results: The unseen-project setting requires strong generalization to new projects and remains an open research problem.This boundary reflects practical use cases in which detection models analyze projects not encountered during training.
4.5 Weighting
The study evaluates weighting strategies for improving generalization across projects. Class-weighted cross-entropy performs best overall, although unseen-project performance remains substantially below seen-project performance and some CWEs are difficult to learn.
- Weighting Results: Class weights for cross-entropy improve generalization for all three tested architectures and achieve the best unseen-project F1 of 17.21% with CodeT5 Small.The comparison includes no weighting, project-balanced sampling, weighted soft F1 loss, and class-weighted cross-entropy.
- Weighting Results: Class weights improve performance on both seen and unseen projects, whereas project-balanced sampling does not improve generalization.Weighted soft F1 loss helps CodeBERT and CodeT5 Small on unseen projects but hurts performance on seen projects.
- Weighting Results: Class weights reduce the seen–unseen F1 gap, but CodeT5 Small still achieves 49.9% on seen projects versus 17.21% on unseen projects.The remaining gap indicates substantial room for improving performance on unknown projects.
- Performance on CWEs: Some CWEs are harder to learn than others regardless of training-data size.CWE-416 illustrates this mismatch: it represents 5.46% of training samples but reaches only 17.86% TPR.
- Performance on CWEs: Very low TPR values for some CWEs are associated with having fewer than 10 vulnerable test samples.Insufficient test samples make those CWE-specific estimates unstable.
5 LABEL ERROR ANALYSIS
The label-error analysis finds that commit-based labeling can mark non-vulnerable or only indirectly related functions as vulnerable. DiverseVul is more accurate than three prior datasets, but its labels remain imperfect.
- Labeling Limitations: The labeling method cannot guarantee that every function changed by a vulnerability-fixing commit is itself vulnerable.This limitation arises because the method labels changed functions as vulnerable based on security issue trackers.
- Analysis Context: CWE-specific TPR estimates can be extremely low when test sets contain too few vulnerable functions.Most CWEs with 0% TPR have fewer than 10 test samples.
- Error Categories: The observed label errors include vulnerabilities spanning multiple functions and changes to non-vulnerable functions relevant or irrelevant to the fix.Examples include adjusting calling parameters or making unrelated whitespace and functionality changes.
- Label Accuracy: DiverseVul’s vulnerable-function labels are 60% accurate, 24 percentage points higher than the combined CVEFixes, BigVul, and CrossVul datasets.Among the prior datasets, CVEFixes is most accurate and BigVul has only 25% label accuracy.
- Error Categories: Irrelevant functions comprise 17.4% to 50% of labeled vulnerable functions across the four analyzed datasets.These functions are labeled vulnerable solely because they were modified in vulnerability-fixing commits.
6 LIMITATIONS
The paper identifies label noise, imperfect de-duplication, and possible test-set contamination as limitations that can affect measured model performance and label accuracy.
- Label noise in DiverseVul and prior datasets may introduce errors into test-set performance measurements.
- 4% of DiverseVul labels and 6% of combined prior-dataset labels were erroneous because whitespace-only changes were treated as security fixes.Whitespace normalization before de-duplication could slightly improve label accuracy.
- Pretraining data may contain test-related code, blog articles, or vulnerability patches, creating unmeasurable contamination risk.The paper cannot rule out effects on results.
- Cloned code may have been modified slightly after inclusion, potentially evading de-duplication and contaminating the test set.
7 CONCLUSION
The paper introduces DiverseVul and uses it with prior datasets to evaluate deep-learning architectures for vulnerability detection. The results support code-specific pretraining and emphasize unresolved generalization and data-scaling questions.
- DiverseVul contains 18,945 vulnerable functions, 330,492 nonvulnerable functions, 155 CWEs, and data from 7,514 commits.It is described as twice the size of CVEFixes and more diverse.
- The study evaluates 11 architectures from four families: GNN, RoBERTa, GPT-2, and T5.
- Increased training-data diversity and volume benefit vulnerability detection, especially for large language models, but the value of still larger datasets remains unclear.
- Code-specific pretraining tasks appear promising for improving deep-learning-based vulnerability detection.
- Generalizing deep-learning models to unknown projects remains a major challenge for future research.
A.1 ReVeal Setup
The ReVeal setup obtains code-property graphs with Joern and specifies training controls for the Gated Graph Neural Network experiments.
- Joern on GitHub is used to obtain Code Property Graphs for the experiments.The authors use a newer version because the older ReVeal version failed to extract almost half of functions into graphs.
- The Gated Graph Neural Network uses different maximum epoch limits for Previous + DiverseVul, Previous, and Section 4.4 experiments.The limits are 50, 100, and 60 epochs, respectively.
- ReVeal training uses Adam with learning rate 0.0001 and weight decay 0.001.
A.2 Fine Tuning Setup
The fine-tuning setup applies classification heads to transformer representations and uses shared optimization and model-selection procedures across language models.
- Classification heads are applied to transformer representations selected by model family: [CLS], last-token, or last-decoder-state embeddings.RoBERTa-family models use [CLS], GPT-2-family models use the last token, and T5-family models use the last decoder state.
- Language models are trained with batch size 32, learning rate 2e-5, Adam, 10 epochs, warmup, and linear learning-rate decay.The best validation model is saved for testing.
- RoBERTa on Previous + DiverseVul with a random split uses the stated learning-rate exception.