Source-linked AI summary
Deep Learning based Vulnerability Detection: Are We There Yet?
Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, Baishakhi Ray
TL;DR
The paper asks whether state-of-the-art DL vulnerability detectors generalize to real-world prediction, where existing techniques are unreliable. It systematically investigates data and model practices, then evaluates a more principled framework that substantially improves performance over prior baselines.
Problem
Existing vulnerability-detection techniques are unreliable, while DL methods reporting up to 95% accuracy have uncertain real-world generalizability.
Method
The paper evaluates four state-of-the-art DL vulnerability-prediction techniques on real-world Chromium and Debian data and develops practices for realistic data collection and semantic model design.
Results
Performance drops by approximately 73% with pretrained models and 54% after retraining, while the proposed practices boost precision by up to 33.57% and recall by 128.38%.
Takeaways & Limitations
Reliable DL vulnerability prediction requires realistic data collection and model design that addresses code semantics, duplication, and class imbalance.
Takeaways & Limitations
Graph-based models can reason about code dependencies but are more expensive than token-based models and may perform poorly in resource-constrained environments.
Abstract
from arXiv · showhide
Automated detection of software vulnerabilities is a fundamental problem in software security. Existing program analysis techniques either suffer from high false positives or false negatives. Recent progress in Deep Learning (DL) has resulted in a surge of interest in applying DL for automated vulnerability detection. Several recent studies have demonstrated promising results achieving an accuracy of up to 95% at detecting vulnerabilities. In this paper, we ask, "how well do the state-of-the-art DL-based techniques perform in a real-world vulnerability prediction scenario?". To our surprise, we find that their performance drops by more than 50%. A systematic investigation of what causes such precipitous performance drop reveals that existing DL-based vulnerability prediction approaches suffer from challenges with the training data (e.g., data duplication, unrealistic distribution of vulnerable classes, etc.) and with the model choices (e.g., simple token-based models). As a result, these approaches often do not learn features related to the actual cause of the vulnerabilities. Instead, they learn unrelated artifacts from the dataset (e.g., specific variable/function names, etc.). Leveraging these empirical findings, we demonstrate how a more principled approach to data collection and model design, based on realistic settings of vulnerability prediction, can lead to better solutions. The resulting tools perform significantly better than the studied baseline: up to 33.57% boost in precision and 128.38% boost in recall compared to the best performing model in the literature. Overall, this paper elucidates existing DL-based vulnerability prediction systems' potential issues and draws a roadmap for future DL-based vulnerability prediction research. In that spirit, we make available all the artifacts supporting our results: https://git.io/Jf6IA.
1 INTRODUCTION
The paper finds that DL-based vulnerability detectors reported high accuracy but perform poorly in realistic settings because dataset construction and model design often fail to capture vulnerability-relevant features.
- Up to 95% accuracy was reported by recent DL vulnerability-detection studies, motivating evaluation of their real-world generalizability.
- Performance drops by approximately 73% when pretrained models are applied directly to real-world data and approximately 54% after retraining.For VulDeePecker, precision falls from 86.9% reported to 11.12% using its pretrained model and 17.68% after retraining.
- Existing approaches are undermined by token-based models, irrelevant learned features, duplicated data, and unrealistic class distributions.Reported training and testing duplication reaches up to 68%, artificially inflating results.
- The paper proposes principled data collection and model design using realistic vulnerability data, semantic representations, de-duplication, and class balancing.Representation learning is used to improve separation between vulnerable and non-vulnerable samples.
- Precision improves by up to 33.57% and recall by up to 128.38% over current baselines after applying these practices.
- The authors curate real-world Chromium and Debian vulnerability data, release artifacts, and characterize DL-based vulnerability detection as an open problem.
2 BACKGROUND AND CHALLENGES
DL-based vulnerability prediction depends on how training data, model representations, and evaluation settings reflect real-world vulnerability detection. The main challenges are unrealistic datasets, limited semantic modeling, class imbalance, and evaluations that may not reveal real-world false positives and false negatives.
- 2.1 DLVP Theory: DLVP models learn vulnerability patterns from labeled code by optimizing parameters to maximize the probability of the observed vulnerable or non-vulnerable labels.The typical pipeline collects and preprocesses data, trains a model, and evaluates it on held-out examples.
- 2.2 Existing Dataset: Existing vulnerability datasets range from synthetic examples and annotations to semi-synthetic data and real code with real-world vulnerability annotations.Synthetic datasets encode known patterns, whereas real datasets include vulnerabilities and fixes from open-source projects.
- 2.2 Existing Dataset: Synthetic and semi-synthetic datasets can omit the semantic complexity of real vulnerabilities, while static-analyzer annotations inherit high false-positive rates.A model trained on simple examples may fail on real vulnerabilities requiring reasoning about multiple code components.
- 2.3 Existing Modeling Approaches: Token-based models lose semantic dependencies because they treat code as a linear token sequence, whereas graph-based models represent syntactic and semantic relationships.Graph-based models can expose data dependencies, but they are more expensive and may perform poorly in resource-constrained environments.
- 2.3 Existing Modeling Approaches: Real-world vulnerability datasets are highly imbalanced, with non-vulnerable code much more frequent than vulnerable code, biasing models toward the majority class.The cited realistic FFMPeg+Qemu dataset instead has an approximately 45%-55% vulnerable/non-vulnerable ratio, which does not reflect real-world distribution.
- 2.4 Existing Evaluation Approaches: Intra-dataset evaluation mainly measures how well a model fits its own dataset and does not comprehensively establish real-world applicability or false-positive and false-negative behavior.False positives and false negatives are directly correlated with developer effort and can discourage model use.
3 REVEAL DATA COLLECTION
REVEAL constructs a real-world vulnerability dataset from Chromium and the Linux Debian Kernel using publicly tracked security fixes. It labels pre-patch functions as vulnerable, post-patch versions as clean, and unchanged functions as clean context.
- Project Selection: REVEAL tracks past vulnerabilities in Chromium and the Linux Debian Kernel, two popular projects spanning browser and operating-system domains.The projects were selected for their large evolutionary histories and diverse security-related program domains.
- Issue Collection: The collection process identifies vulnerability-related issues from Chromium Bugzilla and the Debian security tracker by selecting patches labeled “security”.The identification mechanism is inspired by prior filtering of commits using security-related keywords.
- Function Annotation: For each selected patch, REVEAL extracts changed C/C++ source and header files and labels pre-patch changed functions as vulnerable and fixed versions as clean.Unchanged functions are also included as clean examples, preserving the surrounding real-world code context.
- Real-World Context: The data collection strategy includes both vulnerable and fixed function versions, while retaining unchanged functions from the same files as clean examples.This setup models inspection of a vulnerable function among other functions in scope and may expose the model to the nature of the patch.
4 REVEAL PIPELINE
REVEAL converts source code into graph embeddings, then learns a latent representation designed to separate vulnerable from non-vulnerable examples. Its pipeline combines realistic data balancing with graph-based feature extraction and representation learning.
- Pipeline overview: REVEAL uses two phases: graph-based feature extraction from code, followed by representation learning to distinguish vulnerable and non-vulnerable examples.The first phase produces graph embeddings; the second trains a representation learner on those features.
- Feature extraction: Code property graphs combine control-flow, data-flow, AST, and program-dependency information to represent code syntax and semantics.Each graph contains vertices and edges, with vertex embeddings encoding node types and code fragments.
- Feature extraction: GGNNs update each vertex embedding using information from neighboring vertices, incorporating graph structure into the code representation.The resulting vertex embeddings retain both the original embedding and neighbor information before aggregation into a function-level vector.
- Feature extraction: REVEAL aggregates vertex embeddings into a single graph vector, using element-wise summation as its default configurable aggregation function.The resulting vector is an m-dimensional representation of the original source code.
- Representation learning: REVEAL re-samples imbalanced training data, then trains a representation learner to improve separation between vulnerable and non-vulnerable examples.The representation learner transforms graph embeddings into a latent space and predicts vulnerabilities from that representation.
5 EXPERIMENTAL SETUP
The study evaluates existing vulnerability prediction methods and REVEAL on real-world datasets using repeated randomized splits and multiple classification metrics. It reports robust comparisons with median performance, interquartile ranges, and statistical tests.
- Study subjects: The evaluation compares VulDeePecker, SySeVR, Russell et al., Devign, and REVEAL on two real-world datasets.Devign was reimplemented because its original implementation was unavailable, and the study also used the FFMPeg+Qemu dataset.
- Evaluation formulation: The task is formulated as binary classification of code examples as vulnerable or non-vulnerable.The study focuses on function-granularity vulnerability prediction approaches that take source code as input.
- Evaluation metrics: Accuracy, precision, recall, and F1-score measure classification performance, covering correctness, effectiveness, and combined performance.Precision uses false positives in its denominator, while recall uses false negatives.
- Evaluation procedure: Each experiment runs 30 randomized trials with disjoint 80% training, 10% validation, and 20% test splits.The study reports median performance and interquartile range, and uses significance and effect-size tests for baseline comparisons.
6 EMPIRICAL RESULTS
The empirical study is organized around three questions: the real-world effectiveness of existing approaches, their limitations, and ways to improve DL-based vulnerability prediction.
- Research questions: RQ1 asks how effective existing vulnerability prediction approaches are for real-world vulnerability prediction.It is addressed in Section 6.1.
- Research questions: RQ2 examines the limitations of existing vulnerability prediction approaches.It is addressed in Section 6.2.
- Research questions: RQ3 investigates how DL-based vulnerability prediction approaches can be improved.It is addressed in Section 6.3.
6.1 Effectiveness of existing vulnerability prediction approaches (RQ1)
Existing vulnerability prediction models perform poorly when evaluated on real-world data, both when reused as pretrained models and when retrained. The study reports substantial F1-score declines relative to published baselines.
- Scenario A: pretrained models: Even graph-based Devign reaches only ∼17% F1-score and ∼10% precision on the REVEAL dataset.Similar performance drops are observed for the other baselines.
- Scenario B: retrained models: ∼54% average F1-score drop remains after retraining models with real-world data.On the REVEAL dataset, Russell et al. and VulDeePecker achieve roughly 15% F1-score, while SySeVR achieves 30%.
- Scenario A: pretrained models: ∼73% average F1-score drop occurs when pretrained models are directly applied to real-world vulnerabilities.Existing approaches therefore fail to generalize to real-world vulnerability prediction in this setting.
6.2 Key limitations of existing DLVP approaches (RQ2)
The investigation identifies data duplication, class imbalance, irrelevant feature learning, weak semantic modeling, and poor class separation as key limitations of existing DLVP approaches.
- Data Duplication: Preprocessing techniques introduce duplicates across training and testing data, undermining realistic vulnerability prediction evaluation.Slicing and tokenization can create identical samples from different entry points or different code with the same abstract tokens.
- Data Imbalance: Real-world vulnerability datasets are class-imbalanced because non-vulnerable code substantially outnumbers vulnerable code.This skew can bias models toward the majority class.
- Learning Irrelevant Features: Existing models often learn irrelevant dataset artifacts rather than vulnerability-related code features.Feature-importance analysis is used to inspect which tokens or graph vertices drive predictions.
- Model Selection: Token-based models miss semantic dependencies that graph-based models can use to identify vulnerability-related code.Graph-based models correctly use lines sharing data dependencies with the vulnerable line, whereas token-based models lack this semantic information.
- Model Selection: Existing approaches produce overlapping vulnerable and non-vulnerable representations, making class distinction difficult.The study assesses separation with t-SNE visualizations and centroid distances; larger centroid distances indicate preferable separation.
- Summary: Existing DLVP approaches introduce duplication, mishandle imbalance, fail to learn semantic information, and lack class separability.Addressing these limitations is identified as a path toward improved DLVP performance.
6.3 How to improve DLVP approaches? (RQ3)
REVEAL addresses identified DLVP weaknesses through graph-based feature embedding, data re-balancing, and representation learning. It improves vulnerability-prediction performance across the evaluated datasets and baselines.
- REVEAL Pipeline: REVEAL converts code into graph-based feature vectors, re-balances them with SMOTE, and learns representations that separate vulnerable and non-vulnerable classes.The pipeline uses a code property graph, GGNN, feature-space resampling, and a multilayer perceptron representation learner.
- Overall Performance: On FFMPeg+Qemu, REVEAL’s median accuracy, precision, recall, and F1-score are 5.01%, 5.19%, 13.11%, and 12.64% higher than the next-best approach.The comparison covers all reported performance metrics.
- GGNN: GGNN increases REVEAL’s F1-score on both datasets, with statistically significant improvements and large effect sizes.The effect sizes are 81% for REVEAL and 73% for FFMPeg+Qemu.
- Training Data Balancing: Re-balancing improves performance more on the dataset with greater class skew.SMOTE improves F1-score by 3% on FFMPeg+Qemu and by more than 22% on the REVEAL dataset.
- Representation Learning: The triplet-loss representation learner outperforms Random Forest, MLP, and SVM baselines on both datasets.Median F1-score improvements range from 3.5% to 62.8% on REVEAL and from 6.5% to 23.33% on FFMPeg+Qemu.
- Overall Result: REVEAL improves precision by up to 33.57% and recall by up to 128.38% over state-of-the-art methods.The reported pipeline combines GGNN-based embedding, SMOTE, and representation learning.
7 DISCUSSION
The discussion emphasizes realistic evaluation of vulnerability detection and examines tangled commits as a threat to the REVEAL dataset. An alternate dataset analysis tests whether this threat changes the empirical findings.
- Real-World Evaluation: Real-world evaluation should identify vulnerable functions within an entire codebase rather than rely on artificially balanced test data.The paper proposes an evaluation method intended to simulate this scenario more realistically.
- Real-World Evaluation: Artificially setting the vulnerable fraction to 10% may not reflect real-world imbalance.The paper specifically questions whether Devign’s simulated imbalanced evaluation captures real-world performance.
- Tangled Commits: 80% of vulnerability-fix changes affect 12 or fewer functions, but the collected data remains subject to tangled-commit concerns.Tangled commits combine multiple unrelated or weakly related changes in one commit.
- Robustness Check: After removing patches that change more than one function, REVEAL still outperforms variants without representation learning or data balancing.The alternate dataset reports F1-scores of 26.33%, 22.95%, and 13.13% for the full, no-representation-learning, and no-balancing settings respectively.
8 RELATED WORK
Related work spans machine-learning vulnerability prediction, graph-based program representations, and techniques for mitigating data imbalance. REVEAL builds on these directions by combining graph features, resampling, and representation learning.
- Machine-Learning Vulnerability Prediction: Earlier machine-learning vulnerability studies applied anomaly detection, commit-message analysis, and SVM-based classifiers.The cited work includes multi-class SVM for different vulnerability classes and both static- and dynamic-analysis settings.
- Program Representations: Code Property Graphs combine abstract syntax, control-flow, and program-dependency information for vulnerability reasoning.REVEAL uses CPG-derived graph features in its vulnerability-prediction pipeline.
- Data Balancing: Data-imbalance mitigation methods use different balancing strategies, and REVEAL selects SMOTE for its pipeline.The paper motivates this choice by its reported success in related software-engineering tasks.
9 CONCLUSION
The study identifies shortcomings in existing vulnerability datasets and models that limit practical usability, then introduces a real-world data-collection framework and the configurable REVEAL tool.
- Existing vulnerability datasets are too simple to represent real-world vulnerabilities.
- Existing modeling techniques incompletely address code semantics and data imbalance in vulnerability detection.
- The authors propose a framework for collecting real-world vulnerability data based on these empirical findings.
- REVEAL is a configurable vulnerability prediction tool designed to address concerns identified in existing systems.
- The study demonstrates REVEAL’s potential as a better vulnerability prediction tool for finding real-world vulnerabilities.