Source-linked AI summary
LineVD: Statement-level Vulnerability Detection using Graph Neural Networks
David Hin, Andrey Kan, Huaming Chen, M. Ali Babar
TL;DR
Machine-learning vulnerability detectors commonly predict at the function level, limiting line-specific inspection and interpretation. LineVD directly performs statement-level detection using graph and code representations, and reports a 105% F1-score increase over the current state-of-the-art.
Problem
Function-level vulnerability detectors do not identify the specific contributing lines, while statement-level detection remains insufficiently explored for interpretable predictions.
Method
LineVD formulates statement-level vulnerability detection as binary node classification, combining statement dependencies, function-level information, and source-code representations.
Results
105% increase in F1-score is reported for LineVD over the current state-of-the-art interpretation-based SVD model.
Takeaways & Limitations
LineVD provides statement-level predictions intended to help developers evaluate potentially vulnerable functions more efficiently and achieves reasonable cross-project performance.
Takeaways & Limitations
The dataset is mainly intra-procedural, with limited inter-procedural information, and CodeBERT is not pretrained on C/C++.
Abstract
from arXiv · showhide
Current machine-learning based software vulnerability detection methods are primarily conducted at the function-level. However, a key limitation of these methods is that they do not indicate the specific lines of code contributing to vulnerabilities. This limits the ability of developers to efficiently inspect and interpret the predictions from a learnt model, which is crucial for integrating machine-learning based tools into the software development workflow. Graph-based models have shown promising performance in function-level vulnerability detection, but their capability for statement-level vulnerability detection has not been extensively explored. While interpreting function-level predictions through explainable AI is one promising direction, we herein consider the statement-level software vulnerability detection task from a fully supervised learning perspective. We propose a novel deep learning framework, LineVD, which formulates statement-level vulnerability detection as a node classification task. LineVD leverages control and data dependencies between statements using graph neural networks, and a transformer-based model to encode the raw source code tokens. In particular, by addressing the conflicting outputs between function-level and statement-level information, LineVD significantly improve the prediction performance without vulnerability status for function code. We have conducted extensive experiments against a large-scale collection of real-world C/C++ vulnerabilities obtained from multiple real-world projects, and demonstrate an increase of 105\% in F1-score over the current state-of-the-art.
M. Ali Babar CREST - The Centre for Research on Engineering
The paper lists affiliations with the University of Adelaide and the Cyber Security Cooperative Research Centre.
- The authors are affiliated with the University of Adelaide.
- The listed affiliations connect the work to software technologies and cybersecurity research.
- The paper also lists the Cyber Security Cooperative Research Centre.
CCS CONCEPTS
The paper is categorized under security and privacy, specifically software security engineering.
- The work falls under the Security and privacy research area.
- Its specific topic is software security engineering.
- The classification links the paper's vulnerability-detection focus to software security engineering.
KEYWORDS
The paper concerns software vulnerability detection, program representation, and deep learning, and is published in the MSR 2022 proceedings.
- Software vulnerability detection is a central keyword for the paper.
- Program representation is another keyword, reflecting the paper's focus on representing source code for learning.
- Deep learning is identified as a third keyword and underlies the proposed approach.
- The paper appeared in the Proceedings of the 19th International Conference on Mining Software Repositories in 2022.
1 INTRODUCTION
Existing vulnerability detectors often operate too coarsely for developers to identify vulnerable code, motivating directly supervised statement-level prediction. LineVD addresses this need with a data-driven framework evaluated on realistic vulnerability-detection scenarios.
- 1 INTRODUCTION: Data-driven vulnerability detection has emerged as an alternative to rule-based static analysis because vulnerable patterns are difficult to define comprehensively.
- 1 INTRODUCTION: Function-level and slice-level predictions improve on file-level detection but still provide limited information for validating and interpreting predictions.
- 1 INTRODUCTION: Statement-level predictions can focus developer inspection on suspicious lines and clarify why a function was classified as vulnerable.
- 1 INTRODUCTION: Prior interpretation-based approaches were insufficient for reliably classifying and ranking latent vulnerable statements, motivating direct statement-level training.
- 1 INTRODUCTION: LineVD is proposed as a novel statement-level vulnerability-detection framework with contextual feature extraction and model architectures.
- 1 INTRODUCTION: 105% increase in F1-score is reported for LineVD over the current state-of-the-art interpretation-based vulnerability-detection model.
2 BACKGROUND
Source-code vulnerability detection combines learned token representations with graph structure, including program dependencies that capture relationships between statements.
- Source-code modeling transforms code into vector representations at word, sentence, or document level.
- Prior vulnerability detectors used Doc2Vec, GloVe, and Word2Vec embeddings to represent or aggregate code tokens.
- CodeBERT is a transformer-based bimodal model pretrained on natural-language and programming-language data for source-code understanding.
- Graph representations model source code as nodes and edges, enabling learning of latent relationships beyond token sequences.
- Program dependence graphs are central because software vulnerabilities often involve data and control flows.
3 THE LINEVD FRAMEWORK
LineVD formulates vulnerable-statement identification as binary node classification, combining transformer-based code embeddings with graph attention over program dependencies and joint function-level information.
- 3.1 Problem Definition: LineVD predicts whether each source-code statement is vulnerable by learning a mapping from statement nodes to binary labels.Label 1 denotes vulnerable and 0 denotes non-vulnerable statements.
- 3.1 Problem Definition: Each statement is represented using an n-hop neighborhood graph containing neighboring nodes, dependency edges, and node features.
- 3.2.1 Code Representation: LineVD uses CodeBERT to separately embed the entire function and its individual statements, producing function-level and statement-level representations.The architecture does not aggregate statement embeddings into the function representation.
- 3.2.2 Graph Construction: Graph attention networks propagate information across data- and control-dependent statements while retaining dependency topology.Self-loops include each node in its neighborhood, and two GATs are implemented in the architecture.
- 3.2.3 Classification: A shared linear-and-dropout classifier jointly uses the function-level CodeBERT embedding and statement embeddings produced by GAT layers.The model also incorporates element-wise multiplication to retain prediction consistency.
4 EXPERIMENTAL DESIGN AND SETUP
The evaluation uses realistic, imbalanced vulnerability data with statement-level labels, compares LineVD across embedding and modeling choices, and tests generalization across projects and statement types.
- Research Questions: The study evaluates LineVD through five research questions covering state-of-the-art comparison, embeddings, graph and function information, cross-project testing, and statement types.
- Datasets: Real-world evaluation data should be non-synthetic, imbalanced, sufficiently large, multi-project, and labeled at statement level or traceable to before-fix code.
- Datasets: Big-Vul contains vulnerabilities from over 300 open-source C/C++ projects, spanning 91 vulnerability types and approximately 10,000 vulnerable and 177,000 non-vulnerable samples.
- Ground-truth Labels: Ground-truth vulnerable lines include deleted lines from vulnerability-fixing commits and lines control- or data-dependent on added lines.
- Datasets: Training samples are approximately balanced at the function level, while validation and test sets retain the original imbalanced ratio.
- Evaluation Metrics: The study reports F1, precision, recall, ROC-AUC, PR-AUC, MAP, nDCG, MFR, and N5 interpretation accuracy.PR-AUC is included because it better suits imbalanced problems, while ranked metrics assess confident top-k predictions.
- Experimental Design: Embedding comparisons include CodeBERT, Doc2Vec, and averaged GloVe, while statement-type analysis uses Joern-derived categories.
5 RESULTS
LineVD substantially outperforms IVDetect on ranked and binary statement-level vulnerability detection, while its architecture benefits from code embeddings, graph dependencies, and function-level information. Most predictions rank vulnerable statements highly, although performance varies across statement types and some control-flow statements remain difficult.
- RQ1: Statement-level performance: LineVD significantly outperforms IVDetect across ranked metrics, improving N5 accuracy from 0.695 to 0.900 and MAP@5 from 0.424 to 0.760.The improvements are statistically significant (p < 0.01).
- RQ1: Statement-level performance: Most first-ranking scores fall between 1 and 5, although highly incorrect predictions raise MFR and longer functions may remain challenging.The first-ranking distribution contains 89% of scores between 1 and 5.
- RQ1: Statement-level performance: LineVD outperforms IVDetect by 104% in F1-score and increases recall from 0.140 to 0.533.These binary classification metrics are directly aligned with statement-level vulnerability prediction.
- RQ2: Code embedding methods: CodeBERT improves F1-score over Doc2Vec by 134% and over GloVe by 16% for statement-level vulnerability prediction.The comparisons are statistically significant at p < 0.01 and p < 0.05, respectively.
- RQ3: Graph and function-level information: Using GAT with PDG information increases F1-score from 0.296 to 0.360 compared with no GNN, while GAT performs better than GCN across model types.Using both control and data dependencies generally performs better than using control dependencies alone; adding function-level information with GAT+PDG yields a 140% increase.
- Cross-project evaluation: Performance is generally consistent across projects, though project-based results are slightly lower than random-split results.The Chromium split remains competitive despite accounting for 30% of vulnerable samples.
- Statement-type analysis: LineVD performs best on function declarations and struggles most with continue, break, and goto statements.External function calls also perform worse than built-in calls, while access and cast operations are relatively poorer among operation-related statements.
6 DISCUSSION
The discussion identifies threats from hyperparameter search and dataset context, then outlines performance and architectural limitations that motivate future work.
- Threats to Validity: The dataset primarily captures intra-procedural information, while missing macros and inter-procedural context can limit vulnerability analysis.The authors identify richer contextual datasets as a direction for more comprehensive GNN evaluation.
- Limitations and Future Work: 105% higher F1-score than the current state-of-the-art still leaves substantial room for improvement in statement-level vulnerability detection.The authors specifically identify better code embeddings as a promising direction because CodeBERT was not trained on C/C++.
- Limitations and Future Work: GNN layers inherently propagate information from immediate neighbors, while deeper architectures risk over-squashing and showed no significant gains beyond two layers.This limits the model’s ability to capture longer-range graph information without bottlenecking performance.
- Limitations and Future Work: Future work includes novel GNN architectures and systematic comparisons with other fine-grained vulnerability-detection tools.These directions aim to broaden evaluation and improve vulnerability-pattern capture in program graphs.
7 RELATED WORK
Related work situates LineVD among GNN-based vulnerability detection and interpretable machine-learning approaches. Prior methods commonly operate at function level or derive statement explanations indirectly, leaving direct statement-level classification underexplored.
- GNN on SVD: Existing GNN applications have mainly targeted function-level vulnerability detection rather than direct statement-level classification.The paper frames statement-level GNN capability as insufficiently explored.
- GNN on SVD: GNN-based vulnerability detection uses program graphs to combine syntactic structure with control-flow and dependency information.Code Property Graphs are presented as an influential representation incorporating program dependencies, control flow, and abstract syntax trees.
- GNN on SVD: LineVD addresses this gap by directly training a statement-level SVD framework that combines tokenized node representations with graph-based contextual information.The approach vectorizes source-code lines as graph nodes and uses GNN propagation to exploit relationships among semantically relevant statements.
- Interpretability of SVD Models: Statement-level predictions can improve interpretability by identifying vulnerable lines for developer inspection.Prior work also explores explainable artificial intelligence for line-level defect or vulnerability information.
- Interpretability of SVD Models: Some existing SVD methods provide only partial explanations, such as tokens or intermediate code, rather than complete vulnerable-statement predictions.Another approach localizes statements after function-level vulnerability detection, but detection and localization are often needed together.
- Interpretability of SVD Models: GNN explainers can produce statement-level predictions when graph nodes represent statements, but their suitability as the primary classifier remains unexplored.The paper contrasts post-hoc interpretation of function-level models with directly training on statement-level labels.
8 CONCLUSION
LineVD introduces statement-level vulnerability detection that leverages statement-level information and achieves state-of-the-art performance on real-world open-source projects. Its reasonable cross-project performance indicates effectiveness and generalization to completely unseen software projects.
- LineVD is a deep learning approach for statement-level vulnerability detection that helps developers evaluate potentially vulnerable functions more efficiently.
- LineVD achieves a new state-of-the-art for statement-level vulnerability detection on real-world open-source projects.
- The improvement over the latest fine-grained machine-learning model supports directly utilizing statement-level information for statement-level SVD.
- LineVD achieves reasonable cross-project performance, indicating effectiveness and generalization capabilities on completely unseen software projects.
- Future work includes alternate pretrained feature embeddings and novel GNN architectures better suited to software source code and vulnerabilities.