Source-linked AI summary
Automated software vulnerability detection with machine learning
Jacob A. Harer, Louis Y. Kim, Rebecca L. Russell, Onur Ozdemir, Leonard R. Kosta, Akshay Rangamani, Lei H. Hamilton, Gabriel I. Centeno, Jonathan R. Key, Paul M. Ellingwood, Erik Antelman, Alan Mackay, Marc W. McConley, Jeffrey M. Opper, Peter Chin, Tomo Lazovich
TL;DR
Software vulnerabilities are difficult to detect comprehensively with predefined analysis rules, motivating data-driven learning from open-source C/C++ code. The paper labels functions with static-analysis outputs and compares source-based, build-based, and combined machine-learning models. Source-based approaches outperform build-based approaches, while combining learned deep-model features with a tree classifier gives the strongest reported performance.
Problem
Existing analysis tools detect only limited rule-defined error subsets, motivating learned vulnerability patterns from open-source C/C++ code.
Method
The paper trains function-level machine-learning models on source-code and build-process features, using static-analyzer outputs to label functions.
Results
Source-based approaches outperform build-based approaches, and TextCNN features combined with an extra-trees classifier improve precision-recall performance by 10% over the standard pretrained-word2vec TextCNN.
Takeaways & Limitations
Combining source-based and build-based information can improve detection beyond either model individually, while source code remains the stronger individual approach.
Takeaways & Limitations
The labels treat static-analysis outputs as ground truth, so additional label sources are needed for robust real-world operation.
Abstract
from arXiv · showhide
Thousands of security vulnerabilities are discovered in production software each year, either reported publicly to the Common Vulnerabilities and Exposures database or discovered internally in proprietary code. Vulnerabilities often manifest themselves in subtle ways that are not obvious to code reviewers or the developers themselves. With the wealth of open source code available for analysis, there is an opportunity to learn the patterns of bugs that can lead to security vulnerabilities directly from data. In this paper, we present a data-driven approach to vulnerability detection using machine learning, specifically applied to C and C++ programs. We first compile a large dataset of hundreds of thousands of open-source functions labeled with the outputs of a static analyzer. We then compare methods applied directly to source code with methods applied to artifacts extracted from the build process, finding that source-based models perform better. We also compare the application of deep neural network models with more traditional models such as random forests and find the best performance comes from combining features learned by deep models with tree-based models. Ultimately, our highest performing model achieves an area under the precision-recall curve of 0.49 and an area under the ROC curve of 0.87.
1 Introduction
The paper motivates machine-learning vulnerability detection because existing analysis tools cover limited, rule-defined error subsets, while open-source code enables learning bug patterns directly from data.
- Thousands of vulnerabilities are reported to CVE each year, while additional vulnerabilities are found internally in proprietary software.
- Security vulnerabilities can let attackers force programs into undesired behaviors, including crashes or sensitive-information exposure.
- Existing static and dynamic analysis tools typically detect only a limited subset of errors based on predefined rules.
- The paper applies machine learning to automated vulnerability detection in C and C++ software using patterns learned from open-source repositories.
- Its contribution spans feature-based and source-based detection, traditional and deep-learning methods, and more than fifty CWE weakness categories.
2 Methods
The paper compares source-based and build-based representations for detecting unsafe practices in C and C++ functions, while also evaluating combined models. Source representations use lexed tokens, whereas build representations use compiler-derived program features such as control-flow and instruction behavior.
- Build-based features: Build-based models extract function-level control-flow graphs and basic-block features describing opcodes and variable definition-use relationships.These features are extracted with Clang and LLVM during compilation and build.
- Source-based features: Source-based models represent lexed C/C++ tokens using either bag-of-words frequencies or unsupervised word2vec vectors.The custom lexer categorizes comments, literals, numbers, operators, preprocessor directives, and names; word2vec represents tokens using surrounding context.
- Dataset labeling: Because large ground-truth vulnerability labels are unavailable, the study generates function-level labels by running the Clang static analyzer on source repositories.The approach uses static-analysis outputs rather than manually curated function labels.
- Model comparison: The study constructs separate source-based, build-based, and combined models to compare their vulnerability-detection performance.The combined model concatenates build-based features with a source-based bag-of-words vector.
- Evaluation objectives: The evaluation measures overall prediction of unsafe practices and compares whether build-based or source-based features are more predictive of code quality.The study also considers deep-model and traditional-model configurations, including an extra-trees baseline and a model using convolutional features with extra-trees.
- Scope: The paper notes future work could classify bugs by the type of static-analyzer finding rather than limiting the task to detection.This identifies a scope boundary of the present labeling objective.
3 Experimental results
The experiments evaluate machine-learning classifiers for buggy C/C++ functions across Debian and Github datasets, comparing source-based, build-based, and combined approaches. Source-based models generally outperform build-based models, while combining representations improves performance over either alone.
- 3.1 Datasets: The evaluation uses two open-source C/C++ datasets, with separate training, validation, and test splits and test-only performance reporting.The datasets are Debian packages and Github functions; source-based data include functions that do not need to build successfully.
- 3.1 Datasets: Strict duplicate removal produces conservative estimates by excluding repeated lexed functions and, for build-based data, duplicate feature vectors.The procedure prevents overlap across splits from artificially inflating performance metrics.
- 3.2 Source-based detection: 10% improvement in precision-recall AUC is achieved by feeding TextCNN-learned features into an extra-trees classifier rather than using standard pretrained-word2vec TextCNN input.The two models perform similarly on ROC AUC, while precision and recall are more relevant under severe label imbalance.
- 3.3 Build-based detection: Build-based classifiers generally perform worse than source-based classifiers, while their performance is very similar across the Debian and Github datasets.The authors suggest this similarity indicates the model can learn about different types of code, with cross-dataset generalization left for future work.
- 3.4 Comparison and combined model performance: On the same Github functions, source-based approaches outperform build-based approaches in both ROC AUC and precision-recall space.This apples-to-apples comparison controls for differences in dataset membership, unlike comparisons using the full source dataset.
- 3.4 Comparison and combined model performance: The combined Github model performs better than either the build-only or source-only model, indicating that build-based features add information beyond the source representation.The comparison uses the same set of Github functions for all three model configurations.
4 Conclusions and future work
The paper applies machine learning to detect bugs associated with security vulnerabilities in C/C++ functions using source-code and build-process features. It concludes that the approach can complement static analysis, while noting that robust real-world use requires additional labeling sources.
- 4 Conclusions and future work: Machine-learning models predict static-analysis outputs at the function level using features derived from both source code and the build process.The work covers multiple machine-learning techniques for detecting bugs that can lead to security vulnerabilities in C/C++ code.
- 4 Conclusions and future work: The main limitation is reliance on static-analysis output as ground truth labels, which the authors say should be supplemented for robust real-world operation.Suggested additions include multiple static-analysis tools or a curated large set of programs with known ground truth.
- 4 Conclusions and future work: The authors propose ranking functions by bugginess probability so developers can prioritize code reviews and isolate potential issues.They frame this use as complementing existing static-analysis tools and reducing the time needed to discover potential vulnerabilities.