PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Quality for Software Vulnerability Datasets

Roland Croft, M. Ali Babar, Mehdi Kholoosi

arXiv:2301.05456v1cs.SE

TL;DR

Software vulnerability prediction relies on large datasets, but limited understanding of their quality leaves label reliability and downstream effects insufficiently examined. This study assesses data quality attributes in four datasets and evaluates their impacts on prediction models, finding inaccurate and inconsistent labels that altered performance or hindered training.

  • Problem

    Limited understanding of software vulnerability data quality matters because unreliable labels and biased data can make benchmark results misleading and hinder generalisation to real-world scenarios.

  • Method

    The study measures five data quality attributes using ISO/IEC 25012 across four vulnerability datasets and evaluates attribute impacts with the LineVul prediction model.

  • Results

    20-71% of vulnerability labels were inaccurate, altering performance up to 65%, while inconsistent labels reached 47% and completely hindered training in extreme cases.

  • Takeaways & Limitations

    The findings support systematic data quality assessment and mitigation to improve the validity and trustworthiness of downstream vulnerability prediction tasks.

  • Takeaways & Limitations

    Impact experiments analysed each data quality attribute individually, so confounding from other simultaneous issues remains and cumulative effects require further study.

Abstract

from arXiv · show

The use of learning-based techniques to achieve automated software vulnerability detection has been of longstanding interest within the software security domain. These data-driven solutions are enabled by large software vulnerability datasets used for training and benchmarking. However, we observe that the quality of the data powering these solutions is currently ill-considered, hindering the reliability and value of produced outcomes. Whilst awareness of software vulnerability data preparation challenges is growing, there has been little investigation into the potential negative impacts of software vulnerability data quality. For instance, we lack confirmation that vulnerability labels are correct or consistent. Our study seeks to address such shortcomings by inspecting five inherent data quality attributes for four state-of-the-art software vulnerability datasets and the subsequent impacts that issues can have on software vulnerability prediction models. Surprisingly, we found that all the analyzed datasets exhibit some data quality problems. In particular, we found 20-71% of vulnerability labels to be inaccurate in real-world datasets, and 17-99% of data points were duplicated. We observed that these issues could cause significant impacts on downstream models, either preventing effective model training or inflating benchmark performance. We advocate for the need to overcome such challenges. Our findings will enable better consideration and assessment of software vulnerability data quality in the future.

I. INTRODUCTION

Learning-based software vulnerability prediction depends on large labelled code datasets, yet their quality has received limited systematic attention. The study measures five quality attributes and shows that inaccurate labels and duplication can distort training and benchmarking.

  • Software vulnerability datasets are difficult to construct because labelled examples are scarce, poorly documented, and limited to reported vulnerabilities.Large-scale dataset construction has received more attention than understanding the resulting data quality.
  • Data quality problems can make benchmark results misleading and prevent models from generalising to real-world scenarios.The paper frames these risks as barriers to reliable software assurance.
  • The study systematically analyses accuracy, uniqueness, consistency, completeness, and currentness across vulnerability datasets.For each attribute, it measures prevalence and investigates causes.
  • 20-71% of vulnerability labels were inaccurate in real-world datasets, and up to 47% were inconsistent.These issues cause models to learn false or insufficient patterns.
  • 17-99% of data points were duplicated, and removing duplicates reduced evaluation performance by up to 82%.The authors attribute inflated benchmark results to data leakage caused by duplication.
  • Reliable vulnerability analytics requires addressing data quality, although some issues cannot be remediated through simple rule-based detection and removal.The authors call for knowledge and tools to construct higher-quality datasets.
  • The study contributes methods for measuring software vulnerability dataset quality and directions for mitigating observed issues.A reproduction package is provided to support data quality assessment.

A. Software Vulnerability Data Preparation

Software vulnerability prediction requires large labelled code datasets, but reliable labels are difficult to obtain because vulnerabilities may be unreported and label sources have distinct limitations. Prior data-quality research was limited and non-systematic, motivating quantitative analysis of dataset quality and its effects on prediction models.

  • SVP models require large datasets of code modules labelled as vulnerable or non-vulnerable, but no oracle can unfailingly establish vulnerability presence or absence.Researchers therefore rely on multiple imperfect label sources.
  • Security vendor databases standardise disclosed vulnerability records and often link vulnerabilities to patches and real-world source code.Examples include the National Vulnerability Database and Snyk Vulnerability Database.
  • Developer-provided labels require searching development histories or issue trackers for security-related defects, adding collection effort.This approach supplements vulnerability databases that may incompletely document project vulnerabilities.
  • Tool-created labels expand beyond reported vulnerabilities but depend heavily on static analyser accuracy, which remains contested.Reported vulnerabilities are scarce, while latent or undetected vulnerabilities limit dataset size and add label noise.
  • Synthetic entries ensure label correctness at the cost of source-code diversity.Researchers still face data-preparation challenges regardless of the selected label source.
  • Software engineering datasets collected post-hoc from repositories commonly exhibit accuracy, relevance, and provenance issues.Their data and labels were not explicitly generated for research purposes.
  • Existing data-quality studies are limited and non-systematic, while prior SVP reviews lacked validation against actual datasets and models.This study addresses the gap through quantitative dataset analysis and explicit impact analysis.
  • Automated cleaning frameworks could be developed from a deeper understanding of the quality issues affecting vulnerability datasets.The study’s findings are intended to support such future frameworks.

III. STUDY DESIGN

The study systematically examines data quality in state-of-the-art software vulnerability datasets and evaluates its downstream impact on vulnerability prediction. It analyzes five inherent attributes across four selected datasets using a defined workflow.

  • Research Questions: The study addresses two questions: which data quality issues occur in state-of-the-art vulnerability datasets and how they affect prediction models.
  • Measuring Attributes: The analysis covers accuracy, consistency, completeness, currentness, and uniqueness as inherent data quality attributes.The framework adapts ISO/IEC 25012 by excluding credibility because it is difficult to quantify in these datasets.
  • Dataset Selection: Four datasets were selected to represent different label sources and recent state-of-the-art research: Big-Vul, Devign, D2A, and Juliet.All four datasets provide source code for C/C++ functions and metadata describing label acquisition.
  • Measurement: The study measures each attribute by calculating the proportion of dataset entries satisfying its relevant characteristic.For N samples, dq(i) returns 1 when entry i satisfies the characteristic being measured.

D. Validating Attribute Impact

The study validates data-quality impacts by comparing model performance after mitigation or under altered test setups. It uses LineVul and standard performance metrics across repeated dataset splits.

  • Data-quality impacts were validated by retraining models after mitigation or altering the test setup to isolate the attribute under study.
  • The benchmark used five runs with random 80:10:10 training, validation, and test splits, without preprocessing except removing inconsistent D2A entries.
  • LineVul was selected as the state-of-the-art software vulnerability prediction model for the experiments.LineVul uses CodeBERT representations for lexical and logical code semantics.
  • Performance was evaluated with Recall, Precision, and Matthews Correlation Coefficient, using MCC as the overall indicator.MCC ranges from -1 to 1, with 1 representing optimal performance.

IV. DATA QUALITY ANALYSIS

The data quality analysis finds that vulnerability labels in real-world datasets are constrained by noisy extraction and ambiguous cases. Manual assessment links errors to irrelevant or cleanup changes and inaccurate vulnerability-fix identification, with measurable model-performance effects.

  • Accuracy: Label correctness was assessed manually against each dataset’s labeling mechanism, without verifying whether the represented vulnerability was actually exploitable.The assessment instead examined whether each code snippet was functionally relevant to its reported vulnerability or static-analysis warning.
  • Causes of Inaccuracy: Irrelevant code changes and cleanup changes can cause false-positive vulnerable labels because datasets often treat all code touched by a fix as vulnerable.Refactoring, migration, and facilitating changes may be included even when they do not identify the underlying exploitable code.
  • Limitations: Ambiguous samples were tentatively labeled correct when reviewers could not verify or agree on their labels.The paper identifies unclear vulnerability boundaries and insufficient bug-report information as sources of ambiguity.
  • Dataset Findings: D2A had over two-thirds inaccurate labels, primarily because static-analysis warnings and vulnerability commit identifiers were unreliable.Over 97% of D2A’s included static-analysis warnings came from the lowest reliability class.
  • Model Impact: Incorrect labels reduced precision by 29% for Devign, 50% for Big-Vul, and 80% for D2A under corrected-label evaluation.These decreases were statistically significant according to a Mann-Whitney U test with p < 0.05.

B. Uniqueness

Code duplication occurs across all examined datasets and can bias models or inflate evaluation through cross-set leakage. Its prevalence and causes vary substantially by dataset, with removal sometimes sharply reducing performance.

  • Prevalence: Uniqueness values were 0.830 for Big-Vul, 0.899 for Devign, 0.021 for D2A, and 0.163 for Juliet.Duplication was identified among same-label code clones using a duplicate detector adapted for small functions.
  • Causes: Updated code, similar function sets, and renamed functions were the three main causes of duplication in real-world datasets.These causes reflect spatial and temporal repetition within software repositories.
  • Dataset Findings: D2A contained an average of 57 duplicates per unique function because line-level warnings were abstracted to function scope.Over 94% of D2A entries were type-1 code clones, with additional duplication from forked repositories.
  • Dataset Findings: Juliet also contained many duplicates because small test-case changes preserve repeated code, while fixed statements can be exactly duplicated.
  • Model Impact: Removing duplicates significantly reduced MCC for Devign, D2A, and Juliet, while Big-Vul was not significantly affected.The result indicates that lack of uniqueness is not always problematic.
  • Model Impact: Cross-set duplicates can leak information between training and test data, allowing trivial classification and inflating apparent performance.The analysis focused on evaluation impacts by comparing models with duplicate entries retained or removed.
  • Model Impact: Recall decreased by 7% for Devign, 92% for D2A, and 15% for Juliet after cross-set duplicate removal.Precision increased for Big-Vul and Juliet after removal.

C. Consistency

Consistency requires similar code snippets to have non-conflicting labels, yet all four datasets show inconsistencies arising from dataset-specific collection processes. These issues can distort learned patterns and evaluation, although impacts vary by dataset.

  • Definition and measurement: Consistency means similar code snippets should not receive conflicting vulnerable and non-vulnerable labels.The analysis considers duplicate code entries with conflicting labels, focusing on exact code matches.
  • Observed consistency: 0.999 and 0.991 were the consistency values for Big-Vul and Devign, compared with 0.531 for D2A and 0.75 for Juliet.The authors manually examined 30 inconsistent clusters and found causes that were fairly unique to each collection approach.
  • Causes: Latent vulnerabilities and branch-specific fixing commits produced inconsistent labels in Big-Vul and Devign, respectively.Big-Vul treats untouched functions as non-vulnerable, while Devign can miss fixes present on another code branch.
  • Causes: Repeated static-analysis warnings across edited and unedited commits introduced inconsistent D2A labels, while Juliet duplicates tertiary functions across classes.Juliet’s tertiary functions may be unsafe but exploitable only for particular inputs, producing vulnerable and non-vulnerable copies.
  • Interpretation: Inconsistent labels primarily indicate inaccuracies in non-vulnerable classes formed through the absence of vulnerability labels.The authors identify missing label sources or checks for non-vulnerable code as a central cause.
  • Model impact: Up to 82% lower evaluation performance followed duplicate removal, while consistency changes had smaller or dataset-dependent effects.Inconsistent test entries can produce wrong predictions for identical inputs; MCC increased with consistent test sets, and precision improved more than recall.

D. Completeness

Completeness concerns whether code entries contain all source needed to represent the original function. Missing or truncated code occurred in some datasets but generally had limited measurable impact on model performance.

  • Definition: Completeness was measured as whether source-code entries contained all code from the original function.The analysis treats missing or cut-off function code as incomplete information.
  • Observed completeness: 0.824, 0.944, 0.981, and 1.0 were the completeness values for Big-Vul, Devign, D2A, and Juliet, respectively.Completeness was less frequently problematic than the other examined data-quality attributes.
  • Observed issues: Big-Vul predominantly contained functions truncated at the start, while Big-Vul and Devign also contained prematurely cut-off functions.The authors could not determine the exact truncation cause because dataset-generation scripts were unavailable.
  • Observed issues: D2A contained empty missing values and 13,300 single-line function declarations without functional source code.Empty values arose when static-analysis tools flagged lines outside any containing function.
  • Model impact: Complete training sets showed marginally higher MCC on all datasets, but changes in MCC, precision, or recall were not confirmed as significant.The experiment compared training sets with and without incomplete entries using a common complete test set.
  • Implications: Completeness issues were considered relatively easy to solve and low impact because they affected relatively little information.The paper attributes these issues to data collection.

E. Currentness

Currentness assesses whether a dataset’s temporal distribution matches its application context. The examined datasets showed relatively high currentness, and increasing time between training and testing did not significantly reduce model performance.

  • Definition: Currentness concerns homogeneous temporal characteristics because vulnerabilities and source code evolve over time.The paper relates this issue to concept drift, where input-target relationships change over time.
  • Measurement: Currentness was estimated as one minus Jensen-Shannon divergence between the oldest and newest halves of each dataset.The metric represents the statistical distance between original and current data.
  • Observed currentness: 0.761, 0.811, and 0.844 were the currentness values for Big-Vul, Devign, and D2A, respectively.These values were considered relatively high and unlikely to indicate concept drift.
  • Model impact: No significant decrease in MCC, precision, or recall occurred as the time between training and testing increased.The experiment trained on four earliest partitions, tuned on the fifth, and evaluated on five later partitions; Kendall rank correlation gave p > 0.05.
  • Conclusion: Currentness issues were not observed because the datasets have good temporal distributions collected over long time ranges.Figure 5 depicts the chronological experiment setup used to assess this issue.

V. DISCUSSION

The discussion identifies data quality as a critical but often overshadowed issue in software vulnerability prediction datasets. It recommends checking and remediating duplication, inconsistency, incompleteness, and label inaccuracy while addressing the causes of poor-quality data.

  • Problem: Existing software vulnerability prediction studies often emphasize modeling advances while treating data preprocessing as cursory or inconsistent.The paper’s systematic analysis found critical data issues affecting current datasets.
  • Recommendations: Existing datasets should not be reused without first checking their data quality.The recommendation follows the reported prevalence and downstream effects of quality problems.
  • Recommendations: Inconsistent data points should be removed according to the causes of their inconsistency.The paper links inconsistency to inaccurate non-vulnerable labels and recommends cause-aware remediation.
  • Remediation limits: Rule-based syntactic filters can detect duplication, inconsistency, and incompleteness, but removing noisy samples may leave scarce datasets too small.The authors note that accuracy problems are more severe and lack an existing automatic detection method; inaccuracy could reduce clean entries by a further 20-71%.
  • Root causes: Automatic collection often makes vulnerability labels inaccurate through incorrect vulnerability-fix commits or line changes.The paper identifies semantic filters or heuristics for selecting correct fixing lines as a missing capability.
  • Root causes: Source-code duplication can reduce dataset diversity, motivating more diverse non-vulnerable sampling and improved synthetic-data generation.Bug seeding is promising but still depends on the quality of real-world bugs used to infer seeds.
  • Root causes: Unknown vulnerabilities and undocumented vulnerabilities contribute to inconsistency, especially because non-vulnerable code lacks a reliable label source.The paper mentions semi-supervised semantic filters as a possible way to reduce noise in non-vulnerable labels.

VI. THREATS TO VALIDITY

The study identifies threats involving how data quality attributes are interpreted, how impact experiments are confounded, and how the analysis generalizes across datasets and models.

  • Construct Validity: Construct validity may be limited because the interpreted data quality analysis may not perfectly represent the target attributes.The analysis follows standard practices and existing knowledge, but expert requirements elicitation could improve the claims.
  • Construct Validity: Manual analysis of some attributes may introduce bias or inaccuracies, although two independent raters were used to minimize these effects.
  • Internal Validity: Impact-analysis outcomes may be affected by confounding factors because data quality attributes were analyzed individually while other issues remained present.The authors call for further examination of data quality attributes cumulatively.
  • External Validity: The analysis covers four state-of-the-art datasets with appropriate metadata and evaluates impact using a single state-of-the-art software vulnerability prediction model.The ReVeal dataset could not be investigated because of metadata limitations.

VII. CONCLUSION

The study finds substantial data quality problems in software vulnerability datasets and advocates ongoing, detailed assessment to improve the trustworthiness of downstream data-driven tasks.

  • 20-71% of vulnerability labels were inaccurate in real-world datasets, altering performance by up to 65%.
  • 0-47% of labels were inconsistent, hindering model training completely in the most extreme circumstances.
  • The study systematically examined five data quality attributes across four state-of-the-art software vulnerability datasets.
  • Researchers and practitioners should consider data quality in greater detail and investigate system-dependent attributes for specific operational needs.
  • The authors made their data and analysis scripts available through a reproduction package.
Loading 2301.05456v1…