PaperlayerView exact revision on arXiv

Source-linked AI summary

CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics

Yikun Li, Ting Zhang, Ratnadira Widyasari, Yan Naing Tun, Huu Hung Nguyen, Tan Bui, Ivana Clairine Irsan, Yiran Cheng, Xiang Lan, Han Wei Ang, Frank Liauw, Martin Weyssow, Hong Jin Kang, Eng Lieh Ouh, Lwin Khin Shar, David Lo

arXiv:2411.17274v7cs.SEcs.CR

TL;DR

Existing vulnerability datasets can label unrelated changes in vulnerability-fixing commits as security-related, and NVD-linked function matching does not cover commits without NVD entries. The paper develops VulSifter, an LLM-and-heuristic approach, and CleanVul; GPT-4 with the heuristic achieves an F1-score of 0.82, and CleanVul-trained models show strong results on other datasets.

  • Problem

    Existing vulnerability datasets contain 40%–75% noise from labeling all changes in vulnerability-fixing commits as vulnerability-related, while NVD-linked matching misses commits without corresponding NVD entries.

    Source: §1 · p. 2

  • Method

    VulSifter uses an LLM to analyze function changes with commit messages and context, combined with heuristics that filter unrelated changes, to curate the CleanVul dataset.

    Source: §1 · p. 3

  • Results

    GPT-4 with the heuristic approach achieved an F1-score of 0.82, and CleanVul contains 8,198 function pairs at 90.6% Correctness.

    Source: §10 · p. 21

  • Takeaways & Limitations

    CleanVul-trained models achieved higher accuracy on PrimeVul than models trained and tested exclusively on PrimeVul, and outperformed PrimeVul-trained models on SVEN.

    Source: §1 · p. 3

    The evaluation dataset’s manual labeling is a potential internal-validity threat; the researchers report Cohen’s Kappa agreement of 0.681.

    Source: §8 · p. 19

Abstract

from arXiv · show

Accurate identification of software vulnerabilities is crucial for system integrity. Vulnerability datasets, often derived from the National Vulnerability Database (NVD) or directly from GitHub, are essential for training machine learning models to detect these security flaws. However, these datasets frequently suffer from significant noise, typically 40% to 75%, due primarily to the automatic and indiscriminate labeling of all changes in vulnerability-fixing commits (VFCs) as vulnerability-related. This misclassification occurs because not all changes in a commit aimed at fixing vulnerabilities pertain to security threats; many are routine updates like bug fixes or test improvements. This paper introduces the first methodology that uses the Large Language Model (LLM) with a heuristic enhancement to automatically identify vulnerability-fixing changes from VFCs, achieving an F1-score of 0.82. VulSifter was applied to a large-scale study, where we conducted a crawl of 127,063 repositories on GitHub, resulting in the acquisition of 5,352,105 commits. VulSifter involves utilizing an LLM to comprehend code semantics and contextual information, while applying heuristics to filter out unrelated changes. We then developed CleanVul, a high-quality dataset comprising 8,198 functions using our LLM heuristic enhancement approach, demonstrating Correctness (90.6%) comparable to established datasets such as SVEN and PrimeVul. To evaluate the CleanVul dataset, we conducted experiments focusing on fine-tuning various LLMs on CleanVul and other high-quality datasets. Evaluation results reveal that LLMs fine-tuned on CleanVul not only exhibit enhanced accuracy but also superior generalization capabilities compared to those trained on uncleaned datasets. Specifically, models trained on CleanVul and tested on PrimeVul achieve accuracy higher than those trained and tested exclusively on PrimeVul.

1 Introduction

Vulnerability-fixing commits often mix security fixes with unrelated changes, while existing NVD-linked labeling methods miss commits without NVD entries. The paper introduces VulSifter and CleanVul to filter such changes and evaluates the resulting dataset and models.

1 Introduction

  • Existing vulnerability datasets contain 40%–75% noise because they label every modification in a vulnerability-fixing commit as security-related.A function-name/NVD-description approach [5] does not cover the many GitHub vulnerability-fixing commits without corresponding NVD entries.Source: §1 · p. 2
  • VulSifter improved dataset Correctness from 28.7% to 90.6%, producing CleanVul with 8,198 functions at Correctness comparable to SVEN and PrimeVul.Correctness is the percentage of genuine vulnerable functions in the dataset.Source: §1 · p. 3
  • 58.09% accuracy on PrimeVul with CleanVul-trained CodeBERT exceeded PrimeVul’s own 56.61% intra-dataset accuracy.CleanVul-trained models also achieved 64.87% on SVEN, versus 55.75% for PrimeVul-trained models, and 68.96% when trained and tested on CleanVul.Source: §1 · p. 3
  • Test-related changes and bug fixes make up the predominant non-vulnerability changes, at 41.2% and 38.2%, respectively.The paper’s contributions include a manual characterization of non-vulnerability changes and a dataset evaluation across CleanVul, SVEN, and PrimeVul.Source: §1 · pp. 3–4

2 Motivating Example

Vulnerability-fixing commits can combine security changes with tests, cleanup, dependency updates, and documentation. The examples and analysis illustrate why commits must be examined at the function-change level rather than labeling every modification as a fix.

Motivating Example

  • A ThingsBoard XSS-fix commit combines NoXss validation with unrelated import cleanup, license-documentation updates, and tests.The commit’s stated purpose is XSS mitigation, but not every changed line directly implements the security measure.Source: §2 · p. 4
  • Automatically treating every change in a vulnerability-fixing commit as a security fix can mislabel documentation revisions, code cleanup, and test enhancements.The paper argues that separating direct fixes from other changes can refine vulnerability datasets.Source: §2 · p. 4

Empirical Study

  • A manual analysis of 136 non-vulnerability-related changes used open card sorting to develop a refined taxonomy of function-level change types.Initial categories were drawn from previous research [13].Source: §2 · p. 5
  • Keyword matching and vague commit messages can flag ordinary bug fixes or feature changes as vulnerability fixes, requiring examination of code or issue descriptions.One flagged commit concerned gameplay features rather than a security vulnerability.Source: §2 · p. 5
  • Test-related changes and bug fixes account for 41.2% and 38.2% of non-vulnerability changes in the analyzed vulnerability-fixing commits.Support changes and code refactoring account for 14.7% and 5.1%, respectively; documentation updates account for 0.7%.Source: §2 · pp. 6–7, Table 1 · p. 7

3 VulSifter: Approach

VulSifter analyzes function changes alongside commit messages and other changed functions, then applies heuristics to filter test-related changes. Its scored output allows dataset selection at different confidence levels.

3 VulSifter: Approach

  • VulSifter’s workflow has two stages: LLM Analysis and Heuristics, aimed at identifying function-level vulnerability-fixing changes.The overview figure presents the two-stage framework.Source: §3 · p. 7, Figure 1 · p. 7

LLM Analysis

  • The LLM prompt analyzes function changes with commit messages and other changed functions, assigning a vulnerability-fix confidence score from 0 to 4.Scores range from no vulnerability detected at 0 to very high likelihood at 4.Source: §3 · p. 7, Figure 2 · p. 8

Heuristics

  • Heuristics identify test-related changes from file and function naming conventions and exclude them before LLM processing.For example, function names containing “test” are classified as test-related under the described Pytest naming convention.Source: §3 · p. 8

4 CleanVul: Dataset Curation

CleanVul was curated by applying VulSifter to function changes from GitHub vulnerability-fixing commits. Its score threshold controls the trade-off between dataset size and confidence in vulnerability-fixing changes.

4 CleanVul: Dataset Curation

  • The crawl covered 127,063 repositories and 5,352,105 commits, from which a keyword method identified 43,029 function changes in vulnerability-fixing commits.The analyzed changes spanned Java, Python, C, JavaScript, C#, and C++.Source: §4 · p. 8
  • At threshold 3, VulSifter identified 8,198 vulnerable functions with heuristics and 9,026 without; at threshold 4, it identified 6,368 and 7,020, respectively.The counts decrease at higher thresholds, with additional counts reported for levels 1 and 2.Source: §4 · p. 9, Table 2 · p. 9
  • Threshold 4 represents the highest confidence in vulnerability-fixing changes, while threshold 1 retains broader coverage of potential fixes.Varying thresholds creates datasets at different quality levels.Source: §4 · p. 9

5 Evaluation Settings

The evaluation asks whether VulSifter improves dataset correctness and whether models trained on CleanVul perform well against established datasets and uncleaned data. It compares established datasets and several LLMs using dataset-quality and model-performance metrics.

5 Evaluation Settings

  • The evaluation tests CleanVul’s correctness against uncleaned data, its model performance against established datasets, and the effect of cleaning on model performance.These are the three evaluation questions: RQ1, RQ2, and RQ3.Source: §5 · p. 9

5.1 Comparison with Established Datasets

  • SVEN has 803 functions from manual analysis, while PrimeVul has 6,968 functions but is limited to vulnerabilities linked to NVD descriptions.The paper compares CleanVul with both datasets, which have correctness above 70%.Source: §5.1 · p. 10

5.2 Model Choices

  • The experiments use six encoder-only and decoder-only models, including RoBERTa, CodeBERT, GraphCodeBERT, GPT-2, CodeGPT, and CodeLlama.The study includes three encoder-only models and three decoder-only models.Source: §5.2 · p. 10

5.3 Evaluation Metrics

  • Accuracy, precision, recall, and F1-score assess model performance, while Correctness measures the percentage of genuine vulnerable functions in a dataset.Accuracy is particularly useful here because the paired vulnerable and benign samples are balanced.Source: §5.3 · p. 10

5.4 Implementation Details

  • Fine-tuning uses a 512-token input limit and 50 epochs, with experiments run on NVIDIA H100 GPUs.The implementation uses Huggingface Transformers for fine-tuning and stores optimal checkpoints.Source: §5.4 · p. 11

6 Evaluation Experiments

The experiments find that CleanVul training outperforms uncleaned-data training across same-dataset and cross-dataset tests, while the dataset’s correctness and cross-dataset generalization are also evaluated. Results vary with dataset, language, and training configuration.

6.1 RQ1: What is the correctness improvement of CleanVul compared to the uncleaned dataset?

  • 90.6% Correctness at threshold 3 comes from 8,198 vulnerability-fixing changes, while threshold 4 yields 97.3% Correctness from 6,368 changes.At threshold 3, CleanVul’s Correctness is comparable to SVEN (94.0%) and PrimeVul (86.0%).Source: §6.1 · pp. 11–12, Table 3 · p. 12

6.2 RQ2: How does the performance of LLMs fine-tuned on CleanVul compare to their performance on other established vulnerability datasets?

  • 68.96% accuracy on all-language CleanVul is below SVEN’s 81.86% but above PrimeVul’s 57.19%; Java-only CleanVul reaches 74.78% at best.The all-language result is GraphCodeBERT’s best intra-dataset performance; the Java-only result is the best encoder-only model’s accuracy.Source: §6.2 · p. 14
  • 58.09% vs 57.19% accuracy on PrimeVul favors CleanVul-trained models over PrimeVul-trained models; on SVEN, the corresponding comparison is 64.87% vs 55.75%.Both comparisons report cross-dataset testing; CleanVul training does not exceed SVEN’s same-dataset 81.86% accuracy on SVEN.Source: §6.2 · p. 14
  • 59.02% accuracy from training on Java-only CleanVul and testing on SVEN is comparable to PrimeVul-to-SVEN’s 59.84%, despite the language difference.The paper characterizes this as cross-language generalization from CleanVul’s Java-only training data.Source: §6.2 · pp. 14, 16
  • 55.13% accuracy from training on Java-only CleanVul and testing on PrimeVul is close to PrimeVul’s best 55.26%; CleanVul-trained models also outperform the reverse training direction.The reverse direction’s best reported accuracy is 53.91%, versus 74.78% for training and testing on CleanVul Java only.Source: §6.2 · p. 16

6.3 RQ3: How does the performance of LLMs fine-tuned on CleanVul compare to their performance on the uncleaned dataset?

  • 68.96% vs 55.23% accuracy and 71.57% vs 46.32% F1-score on the same dataset favor CleanVul over uncleaned-data training; CleanVul also scores higher on PrimeVul and SVEN.On PrimeVul, CleanVul versus uncleaned training yields 54.65% vs 52.42% accuracy and 59.59% vs 40.50% F1-score; on SVEN, the values are 62.07% vs 52.19% and 69.88% vs 54.33%, respectively.Source: Table 6 · p. 17, §6.3 · p. 17

7 Additional Analyses

The additional analyses assess LLM sensitivity using manually labeled function changes and test how VulSifter’s inputs, heuristic filtering, and scoring design affect performance.

7.1 Sensitivity Analyses

  • 775 function changes were manually analyzed, with two independent researchers reviewing each change after three rounds and agreement reaching Kappa 0.681.The authors report that this sample exceeds the statistically significant sample size of 385 for a 95% confidence level and 5% margin of error.Source: §7.1 · pp. 17–18
  • 82.24% F1-score: GPT4 led the evaluated models in identifying vulnerability-fixing changes.GPT3.5 scored 75.85% and GPT4o 74.73%.Source: Table 7 · p. 18, §7.1 · p. 18

7.2 Ablation Studies

  • Removing contextual information reduced VulSifter’s F1-score from 82.24% to 77.64%, a decrease of 4.6%.Context here is information about other changed functions in the commit.Source: §7.2 · p. 18
  • Removing commit messages reduced F1-score from 82.24% to 81.17%, a decrease of 1.07%.The study describes commit messages as potentially informative about the nature and purpose of changes.Source: §7.2 · pp. 18–19
  • Removing both commit messages and context reduced F1-score to 76.00%, a decrease of 6.24% from the full configuration.This variant relies solely on function changes.Source: §7.2 · p. 19
  • Replacing the 0-4 scoring output with binary output reduced F1-score from 82.24% to 74.66%, a reduction of 7.58%.This comparison used the best-performing input combination: function changes, commit message, and context.Source: §7.2 · p. 19, Table 9 · p. 19
  • Removing heuristics that filter test-related changes reduced F1-score from 82.24% to 78.97%, a reduction of 3.27%.The authors describe these heuristics as filtering test-related changes before processing.Source: §7.2 · p. 19, Table 10 · p. 19

8 Threats to Validity

The study identifies potential validity threats from manual evaluation labels and probabilistic LLM behavior, while describing steps taken to address them and the scope of its data.

Internal Validity

  • Manual labeling may threaten internal validity; multiple researchers labeled the data, with Cohen’s Kappa of 0.681 indicating substantial inter-rater reliability.The authors also report multiple analysis rounds and cross-validation among researchers.Source: §8 · p. 19
  • Probabilistic LLM behavior can cause performance variation between runs, so each experiment was run three times and the average reported.Source: §8 · p. 20

External Validity

  • The dataset includes Java, Python, C, JavaScript, C#, and C++ code, alongside projects with diverse development practices and team sizes.The authors present this diversity as supporting applicability across languages and typical software development environments.Source: §8 · p. 20

Construct Validity

  • Evaluation used standard vulnerability-detection metrics: F1-score, precision, recall, accuracy, and correctness.Source: §8 · p. 20

9 Related Work

Related work covers vulnerability datasets, automated detection, and dataset cleaning; the paper positions CleanVul and VulSifter as addressing gaps in dataset quality and coverage.

9.1 Vulnerability Datasets

  • CleanVul reports 90.6% correctness across 8,198 functions, while SVEN reports 94% across 803 functions and PrimeVul reports 86% [10] [5].BigVul and CodeXGLUE [17] [26] exceed 10K vulnerable functions but have correctness rates around 25% [4]; CrossVul, CVEFixes [1], and DiverseVul [4] report 47.8%, 51.7%, and 60%, respectively.Source: §9.1 · p. 20

9.2 Automated Vulnerability Detection Techniques

  • Prior automated vulnerability detection spans static analysis, supervised learning, deep belief networks, VulDeePecker [14], graph neural networks [26], and transformers [4].The paper notes that machine-learning methods depend heavily on training-data quality.Source: §9.2 · pp. 20–21

9.3 Dataset Cleaning and Noise Reduction

  • Ding et al. [5] report 40% to 75% noisy data in existing vulnerability datasets and use function-name matching with NVD descriptions; Chen et al. [4] use multi-stage filtering.Source: §9.3 · p. 21
  • VulSifter is presented as the first automatic approach to filter non-vulnerability changes without requiring NVD links, and CleanVul is derived from GitHub data.The paper describes CleanVul as complementing existing high-quality datasets.Source: §9.1 · p. 20, §9.3 · p. 21

10 Conclusion and Future Work

The paper concludes that VulSifter identifies genuine vulnerability fixes and supports CleanVul’s construction from GitHub commits. CleanVul-trained models show cross-dataset performance, and the authors outline future work on long-form artifacts, specialized models, context integration, and curation.

10 Conclusion

  • 41.2% of non-vulnerability changes were test-related and 38.2% were bug fixes; GPT-4 with VulSifter achieved an F1-score of 0.82 without requiring NVD links.Source: §10 · p. 21
  • CleanVul contains 8,198 vulnerable function pairs at threshold 3 with 90.6% Correctness; threshold 4 yields 6,368 pairs with 97.3% Correctness in the test sample.The study analyzed 5,352,105 commits from 127,063 GitHub repositories.Source: §10 · p. 21
  • On PrimeVul, CleanVul-trained models achieved 58.09% accuracy, versus 57.19% for models trained and tested on PrimeVul.CleanVul-trained models also scored 64.87% versus 55.75% when tested on SVEN; on PrimeVul, CleanVul- and SVEN-trained models scored 58.09% and 55.20%, respectively.Source: §10 · p. 21

Future Work

  • Future work targets hierarchical analysis of long-form artifacts, specialized vulnerability-detection models, intelligent context integration, and semi-automated curation with expert validation.Source: §10 · p. 22

A Appendix

The appendix identifies test functions and files across five programming languages using regular-expression patterns. It also shows the prompt's binary classification output for whether changes are vulnerability-fixing.

A Appendix

  • Fig. 3 gives regular-expression patterns for identifying test functions and test files in Java, C++, C#, JavaScript, and Python.The patterns capture function declarations and test-file naming conventions used in testing frameworks.Source: Figure 3 · p. 24
  • The paper lists receipt on 20 February 2007, revision on 12 March 2009, and acceptance on 5 June 2009.Source: A Appendix · p. 24
  • Fig. 4 shows that the prompt produces a binary output indicating whether changes are vulnerability-fixing.Source: Figure 4 · p. 25
Loading 2411.17274v7…