An Empirical Study of Deep Learning Models for Vulnerability Detection

Benjamin Steenhoek; Md Mahbubur Rahman; Richard Jiles; Wei Le

Source-linked AI summary

An Empirical Study of Deep Learning Models for Vulnerability Detection

Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, Wei Le

arXiv:2212.08109v3Dec 15, 2022cs.SEcs.CRcs.LG

TL;DR

Deep learning vulnerability-detection models show promise, but their behavior and data requirements remain insufficiently understood. The paper surveys and reproduces state-of-the-art models on Devign and MSR, finding substantial run-to-run variability, low cross-model agreement, and limited gains from more data. These findings support closer examination of model robustness, training data, and interpretations, while remaining bounded by the representativeness of the studied models and datasets.

Problem
Deep learning vulnerability-detection models are promising but insufficiently understood, including their capabilities, training data needs, and decision features.
Method
The authors surveyed and reproduced state-of-the-art models, then studied 6 research questions across model capabilities, training data, and model interpretation using Devign and MSR.
Results
The study found substantial prediction variability across runs, low agreement across models, limited performance gains from larger datasets, and shared model features identified by explanation tools.
Takeaways & Limitations
The findings provide lessons and guidelines for understanding model results, preparing training data, and designing and debugging future vulnerability-detection models.
Takeaways & Limitations
The observations may not generalize beyond the available models and datasets, which may not represent real-world vulnerability detection.

Abstract

from arXiv · show

Deep learning (DL) models of code have recently reported great progress for vulnerability detection. In some cases, DL-based models have outperformed static analysis tools. Although many great models have been proposed, we do not yet have a good understanding of these models. This limits the further advancement of model robustness, debugging, and deployment for the vulnerability detection. In this paper, we surveyed and reproduced 9 state-of-the-art (SOTA) deep learning models on 2 widely used vulnerability detection datasets: Devign and MSR. We investigated 6 research questions in three areas, namely model capabilities, training data, and model interpretation. We experimentally demonstrated the variability between different runs of a model and the low agreement among different models' outputs. We investigated models trained for specific types of vulnerabilities compared to a model that is trained on all the vulnerabilities at once. We explored the types of programs DL may consider "hard" to handle. We investigated the relations of training data sizes and training data composition with model performance. Finally, we studied model interpretations and analyzed important features that the models used to make predictions. We believe that our findings can help better understand model results, provide guidance on preparing training data, and improve the robustness of the models. All of our datasets, code, and results are available at https://doi.org/10.6084/m9.figshare.20791240.

I. INTRODUCTION

The paper frames deep learning vulnerability detection as promising but insufficiently understood, motivating a systematic study of model capabilities, training data, and interpretation.

Deep learning vulnerability detection has reported promising results, including 0.9 F1 scores and performance exceeding static analyzers.These results have attracted substantial industrial interest and investment.
Existing research has focused mainly on applying emerging models to datasets, leaving model behavior, suitable vulnerability granularity, training data, and used information poorly understood.The authors identify these questions as difficult because deep learning models are black boxes.
Model capabilities: The study examines whether model outputs vary across runs and models, and whether vulnerability-specific models should replace one model covering all vulnerability types.
Training data: The study investigates how training-data size and project composition affect vulnerability-detection performance.
Model interpretation: The study uses explanation tools to examine which source-code information models use and whether models agree on important features.
The authors survey models, reproduce 11 frameworks, compare 9 models across Devign and MSR, and design 6 research questions to guide model understanding.

II. A SURVEY OF MODELS AND THEIR REPRODUCTION

The authors surveyed and reproduced state-of-the-art vulnerability-detection models spanning multiple architectures, then standardized 9 models for studies on Devign and MSR.

The survey covered models from 2018 to 2022 and reproduced 11 open-source models identified through research papers and two leaderboards.The complete model list and reproduction failures are documented in the replication package.
The reproduced models span GNN, MLP, RNN, Bi-LSTM, CNN, and Transformer architectures operating on graphs, tokens, ASTs, sequences, or code representations.
Devign contains 27,318 roughly balanced examples, whereas MSR contains 10,900 vulnerable and 177,736 non-vulnerable examples labeled by projects and CWE entries.These dataset properties support analyses involving class balance, project composition, and vulnerability type.
Reproduction results generally fell within 2% of the original papers when using their original datasets and settings.ReVeal reproduction fixed a data-leakage error confirmed by its authors, while Devign used a third-party reproduction because original code was unavailable.
The authors adapted implementations to both Devign and MSR, excluding VulDeeLocator and SySeVR because they could not be easily modified.The resulting comparative studies used the remaining 9 models.

III. RESEARCH QUESTIONS AND FINDINGS

The paper organizes its empirical investigation into model capabilities, training data, and model interpretation, with each research question evaluated through a defined setup and findings.

The research questions are grouped into model capabilities, training data, and model interpretation.
The study reports reproduction results in Table II using accuracy, precision, recall, and F1 metrics, averaged over three random seeds.
For each research question, the paper presents motivation, study setup, and findings.

A. Capabilities of Deep Learning Models

The study finds substantial variability across training runs and limited agreement across models, while vulnerability type, code features, and example difficulty shape detection performance. Models generally perform better on vulnerability-specific data and easy examples, but cross-bugtype generalization is weaker.

Run variability: 34.9% of test data changed predictions across random seeds, while F1 scores varied by 2.9 on average.ReVeal showed the highest variability, with outputs changing for 50% of test data.
Model agreement: Only 7% of test examples received the same prediction from all models, limiting differential testing without ground-truth labels.Agreement was higher among selected architecture groups but remained incomplete.
Vulnerability types: Input validation and Resource errors often had lower performance, whereas Buffer overflows and Value errors often performed better.Resource errors may be difficult when relevant code is separated by long-range dependencies or when training data covers varied resources incompletely.
Vulnerability types: Vulnerability-specific models generally outperformed combined models, although combined training sometimes improved Input validation and Resource error detection.Cross-bugtype performance was usually much lower than same-bugtype performance, suggesting differing vulnerability data distributions.
Example difficulty: All 9 models performed better on easy than difficult examples, with an average performance gap of 10.3%.The original test-set performance fell between the easy and difficult sets for 7 of 9 models.
Model interpretation: Control-flow features varied across models, while call, length, and pointer features had near-zero coefficients across models.The feature if had a negative coefficient for every model, and arrays and switch were associated with positive coefficients.

B. The Training Data

The studies examine whether training-data size and project composition affect vulnerability-detection performance. More data generally helps, but gains vary by model, dataset balance, and whether projects overlap between training and testing.

Dataset size: Training-data size was studied using combined Devign and MSR datasets, with 10% held out for testing and training subsets ranging from 1% to 100%.The imbalanced dataset contained 194,285 examples, while the balanced dataset contained 45,363 examples.
Dataset size: Generally, models improved as more data were added, but the improvement was not significant and performance often fluctuated.On the balanced dataset, average F1 showed no difference between 10% and 100% of the data; on the imbalanced dataset, average F1 improved by 0.16.
Dataset size: Some models reached good performance with only 5% of balanced training data, whereas ReGVD and CodeBERT required 50% and 30% of imbalanced data, respectively.The 5% balanced subset contained about 2,268 examples, including 1,134 vulnerable examples.
Project composition: Using diverse projects did not benefit the models compared with training only on Chrome; 5 of 6 models had higher median performance on non-diverse data.The comparison used 76k-example training sets and five-fold cross-validation.
Project composition: Mixed-project training and testing outperformed cross-project evaluation for every model, with average and maximum F1 gaps of 0.11 and 0.32.The result indicates that project overlap can improve predictions on data from the same project, while unseen-project evaluation is harder.

C. Internals of Deep Learning

The interpretation studies examine which code features vulnerability-detection models use and how consistently they use them. Models often emphasize recognizable code patterns, but their explanations can omit causal dependencies and reflect input-length constraints.

Interpretation setup: The study used GNNExplainer for Devign and ReGVD and LIT for LineVul, VulBERTa-CNN, VulBERTa-MLP, CodeBERT, and PLBART.These tools were selected because they supported the largest number of investigated models.
Interpretation setup: Important features were normalized into source-code lines, and each example’s top 10 highest-scored lines formed its important feature set.GNNExplainer scores graph edges and LIT scores tokens; line scores were computed from the corresponding token or node scores.
Model agreement: Feature-set overlap was highest between LineVul and ReGVD, which shared 7.29 lines for corrected examples.Similarity was measured using feature-set intersection and the Jaccard index.
Highlighted code features: Models commonly highlighted for, if, while, function signatures, memory operations, and error-reporting code as important features.The study confirmed these patterns through keyword profiling and reported them in Table VII.
Highlighted code features: Important feature sets contained error lines at 2.79 times the probability observed across the program on average.The comparison used the probability of error in the important feature set versus the probability in the full function or program.
Interpretation findings: Transformer models sometimes predicted correctly despite truncated inputs that omitted the vulnerability’s root cause, while missed bugs were often application-specific and potentially underrepresented in training data.The authors also found that models can rely on vulnerability patterns rather than reasoning about values and long-range semantic dependencies.

IV. THREATS TO VALIDITY

The study’s observations are constrained by the available models and datasets, which may not represent real-world vulnerability distributions. Additional threats include subjective vulnerability grouping and imperfect interpretation tools.

Dataset and model scope: The findings may not generalize beyond the available deep learning models and vulnerability-detection datasets.The authors used balanced Devign and imbalanced MSR datasets, but both may still fail to represent real-world vulnerability distributions.
Experimental design: Vulnerability-type grouping in RQ2 is subject to researcher judgment and may introduce bias.Two domain-knowledgeable authors independently inspected the CWE list, then discussed and agreed on the grouping.
Experimental design: RQ6 interpretations may be imperfect because the selected tools cannot always identify the features models use.Experiments requiring customized data also depended on models working outside their shipped datasets, and problematic models were excluded when tuning failed.

V. RELATED WORK

Prior work has examined vulnerability-detection models, architectures, datasets, and evaluation settings, but this study positions itself as the first to characterize programs and code features that models predict poorly.

Existing empirical studies: Earlier empirical studies examined synthetic data, duplication, imbalance, architectures, representations, and symbolization across several vulnerability-detection models.Chakraborty et al. studied four models, while Tang et al. surveyed two models to compare design choices.
Model architectures: Vulnerability-detection research has explored MLP, RNN, CNN, Transformer, and GNN architectures with varied representations and training strategies.Examples include Devign’s gated graph neural network, LineVul’s pretrained Transformer, and ReVeal’s use of SMOTE for imbalance.
Evaluation settings: Most prior models were evaluated on indistribution data, while some studies examined vulnerability-specific difficulty or cross-project performance.Indistribution settings may overlap projects and bug types between training and testing; other work found some vulnerabilities harder or evaluated held-out projects.
Research gap: This work is presented as the first attempt to characterize programs and code features that vulnerability-detection models cannot predict well.The characterization targets the programs and features associated with poor prediction rather than only comparing model architectures.

VI. CONCLUSIONS AND FUTURE WORK

The study evaluates deep learning vulnerability-detection models across six research questions and reports findings about prediction stability, vulnerability scope, data, difficult programs, and model explanations. It concludes with recurring patterns in model behavior and future investigation of highlighted code patterns.

Model capabilities: 34.9% of test data received different predictions across runs, while only 7% of predictions agreed across nine models.These results demonstrate substantial variability within models and low agreement between models.
Model capabilities: Models trained for a specific vulnerability type generally performed better than models trained for all vulnerabilities.The conclusion compares vulnerability-specific training with an all-vulnerability model.
Training data: Performance did not increase significantly with larger datasets, and models began performing well with around 1k vulnerable examples.This pattern held for both balanced and imbalanced datasets.
Difficult programs: A logistic regression model identified programs that were difficult for deep learning models to predict correctly.The study used this model to relate code features to prediction difficulty.
Model interpretation: Interpretation tools found 3.38-6.88 lines in common among the top 10 important lines identified by different models.The study also reported code patterns that models frequently highlighted as important features.
Future work: Future work will further investigate the code patterns that models frequently highlighted as important features.The conclusion identifies these recurring patterns as the next research target.