Source-linked AI summary
Automated Vulnerability Detection in Source Code Using Deep Representation Learning
Rebecca L. Russell, Louis Kim, Lei H. Hamilton, Tomo Lazovich, Jacob A. Harer, Onur Ozdemir, Paul M. Ellingwood, Marc W. McConley
TL;DR
Software vulnerabilities remain difficult to detect at scale because existing tools cover limited rule-defined patterns and prior learning studies used limited data. This paper builds a large function-level C/C++ dataset, learns representations from lexed source code, and combines neural features with ensemble classification. The resulting approach performs strongly across natural and benchmark code, supporting deep representation learning as a promising vulnerability-detection strategy.
Problem
Existing analysis tools detect limited rule-defined error subsets, while prior learning-based vulnerability detection used datasets with limited size and variety.
Method
The study mines millions of C/C++ functions, labels natural code with curated static-analyzer findings, and classifies lexed source-code representations using neural features and ensemble trees.
Results
The best overall results use convolutional neural network features classified with an ensemble tree algorithm, with CNN-based approaches outperforming RNNs and neural-feature random forests outperforming standalone networks and BOW.
Takeaways & Limitations
Deep feature representation learning directly from source code is a promising approach for scalable automated software-vulnerability detection.
Abstract
from arXiv · showhide
Increasing numbers of software vulnerabilities are discovered every year whether they are reported publicly or discovered internally in proprietary code. These vulnerabilities can pose serious risk of exploit and result in system compromise, information leaks, or denial of service. We leveraged the wealth of C and C++ open-source code available to develop a large-scale function-level vulnerability detection system using machine learning. To supplement existing labeled vulnerability datasets, we compiled a vast dataset of millions of open-source functions and labeled it with carefully-selected findings from three different static analyzers that indicate potential exploits. The labeled dataset is available at: https://osf.io/d45bw/. Using these datasets, we developed a fast and scalable vulnerability detection tool based on deep feature representation learning that directly interprets lexed source code. We evaluated our tool on code from both real software packages and the NIST SATE IV benchmark dataset. Our results demonstrate that deep feature representation learning on source code is a promising approach for automated software vulnerability detection.
I. INTRODUCTION
Software vulnerabilities arise from subtle programming errors and can compromise systems, while existing analysis tools detect only limited rule-defined subsets. The paper addresses this gap with machine learning for automated vulnerability detection directly from real-world C/C++ source code.
- Subtle software errors can create vulnerabilities that enable system compromise, information leaks, denial of service, and other damaging effects.
- Open-source repositories provide data for learning vulnerability patterns directly from code rather than relying only on predefined analysis rules.
- The paper presents machine-learning techniques for automated vulnerability detection learned from real-world C/C++ code examples.
- Existing static and dynamic analysis tools are constrained by hand-engineered rules and cannot guarantee full codebase coverage.
- Prior vulnerability-detection studies used limited datasets or representations, restricting the scale and variety available to deep learning.
III. DATA
The study constructs a large function-level C/C++ dataset from synthetic and natural-code sources because effective vulnerability learning requires many examples spanning complex and varied programs.
- The dataset contains over 12 million function-level C/C++ examples mined from Juliet, Debian, and public GitHub repositories.
- Functions are analyzed at function level because it is the lowest granularity that captures a subroutine’s overall flow.
- Table I summarizes functions collected, retained after filtering, and categorized as vulnerable or not vulnerable across the data sources.
- Juliet supplies labeled examples from 118 CWE classes but is synthetic and insufficiently representative of natural code on its own.
A. Source lexing
The data pipeline standardizes source code into compact lexed representations and removes duplicate or near-duplicate functions to reduce vocabulary complexity and limit leakage into evaluation.
- A. Source lexing: The custom lexer captures critical-token meaning while keeping representations generic and vocabulary size small.
- A. Source lexing: The lexer uses a 156-token vocabulary, strips comments, and replaces literals and identifiers with type-specific representations.
- A. Source lexing: Duplicate and near-duplicate functions can inflate metrics by allowing training information to leak into test sets.
- A. Source lexing: Duplicates were removed using identical lexed representations or compile-level feature vectors based on control flow, operations, and variable use-def information.
- B. Data curation: About 10.8% of pulled functions remained after strict duplicate removal and cuts, providing a conservative estimate of performance on unseen code.
C. Labels
Because mined open-source functions lack known ground truth, the study evaluates several labeling strategies and ultimately uses curated, vulnerability-relevant findings from multiple static analyzers.
- C. Labels: Most mined open-source functions lacked known ground-truth labels, making function-level vulnerability labeling a substantial challenge.
- C. Labels: Dynamic analysis was too resource-intensive for the dataset, taking nearly a day for roughly 400 functions in one LibTIFF module.
- C. Labels: Commit-message labeling produced low-quality labels, while keyword-based tagging reduced candidates and still required substantial manual inspection.
- C. Labels: Labels were generated with Clang, Cppcheck, and Flawfinder, whose outputs differ in search scope and detected finding types.
- C. Labels: Security researchers mapped analyzer findings to CWEs and converted selected findings into binary vulnerable or not-vulnerable labels.
- C. Labels: 149 of 390 analyzer finding types were judged potentially security-relevant, and roughly 6.8% of curated mined functions triggered such findings.
IV. METHODS
The primary approach combines neural feature representations of lexed function source code with a random forest ensemble classifier for vulnerability detection.
- The system combines neural representations of lexed function source code with a random forest classifier.
A. Neural network classification and representation learning
The method applies NLP-inspired CNN and RNN feature extraction to lexed source code, learning token embeddings during classification and improving resistance to overfitting with Gaussian noise.
- CNNs and RNNs extract features from embedded source representations for function-level vulnerability classification.
- 1) Embedding:: Tokens are embedded into learned fixed k-dimensional vectors during classification training via backpropagation.Unsupervised word2vec initialization yielded minimal improvement over randomly initialized learned embeddings.
- The architecture lexes variable-length source code, embeds it, applies convolutions, maxpools the sequence, and feeds the fixed-size vector to a random forest.The embedding and convolutional filters are learned using weighted cross entropy through fully connected classification layers.
- 1) Embedding:: Random Gaussian noise added to each embedded representation substantially improved resistance to overfitting.It was more effective than regularization techniques such as weight decay.
2) Feature extraction:
Feature extraction uses convolutional and recurrent layers followed by max pooling, producing fixed-size representations while preserving vulnerability signals found anywhere in variable-length functions.
- 2) Feature extraction:: Convolutional filters span the full token-embedding space, and filter size m = 9 with n = 512 filters worked best.Batch normalization followed by ReLU was most effective.
- 2) Feature extraction:: Max pooling along sequence length converts variable-length convolutional or recurrent features into fixed-size representations.The resulting representation has size n for convolutional features or n′ for recurrent features.
4) Dense layers:
The classifier uses dropout and two dense hidden layers, while training restricts function lengths, addresses class imbalance, and optimizes CNNs and RNNs with cross entropy.
- 4) Dense layers:: Two hidden layers of 64 and 16 units before the final softmax output layer gave the best classification performance.Training used 50% dropout on connections from the maxpooled representation to the first hidden layer.
- 4) Dense layers:: Training used functions with token lengths 10 ≤ ℓ ≤ 500, padded to length 500, with batch size 128 and Adam optimization.CNN and RNN learning rates were 5 × 10^-4 and 1 × 10^-4, respectively, with cross entropy loss.
- 4) Dense layers:: Vulnerable functions received greater loss weight because the dataset was strongly unbalanced.
B. Ensemble learning on neural representations
Neural networks automatically learn features, but their standalone classification was suboptimal on the full dataset. Feeding neural representations into ensemble classifiers produced the best results and facilitated retraining.
- Neural network standalone classification was suboptimal on the full dataset.
- Random forest and extremely randomized trees achieved better results when trained on neural features from CNNs or RNNs.The features came from sequence-maxpooled convolution or recurrent outputs.
- Separately optimizing neural features and the classifier appeared to help resist overfitting.
- The ensemble design also made it easier to retrain classifiers on new feature sets or feature combinations.
V. RESULTS
The results compare neural and ensemble models across natural Debian/GitHub functions and labeled SATE IV data, using precision-recall, ROC, MCC, F1, and vulnerability-type analyses. CNN-based neural representations with random forests performed strongly, while performance varied across vulnerability types and datasets.
- The bag-of-words benchmark ignores token order and exploits correlations with source length, complexity, and commonly misused call combinations.Improvements over this baseline indicate more complex and specific vulnerability patterns.
- CNN models outperformed RNN models as standalone classifiers and feature generators while training faster with fewer parameters.
- Random forests trained on neural representations outperformed the corresponding standalone networks and the benchmark bag-of-words classifier.
- Precision-recall, ROC AUC, MCC, and F1 evaluated primary ML approaches on the natural Debian and GitHub function test data.Figure 2 shows precision-recall performance; Table III reports the listed metrics at validation-optimal thresholds.
- Some vulnerability types were significantly more challenging than others for the multi-label CNN + RF classifier.
- On SATE IV, ML models, especially CNNs, performed much better than on natural Debian and GitHub functions, while Clang found few vulnerabilities compared with ML methods.The authors attribute the stronger SATE IV performance likely to more examples per vulnerability and more consistent style and structure.
- ML methods rapidly score large uncompiled repositories and expose tunable probability thresholds, whereas static analyzers return fixed numbers of findings.Feature activation maps can help explain ML decisions, although static analyzers localize findings better.
VI. CONCLUSIONS
The paper demonstrates machine-learning-based vulnerability detection directly from C/C++ source code using curated datasets and learned source representations. It identifies improved labeling and larger datasets as priorities for making this approach more broadly practical.
- The system combines a large C/C++ dataset from Debian and GitHub, curated static-analysis findings, and the SATE IV dataset.The dataset supports function-level vulnerability detection using real-world source code and benchmark data.
- A custom lexer produces generic function-source representations, while a convolutional neural network and ensemble tree classifier achieve the best overall results among the evaluated techniques.The methods were adapted from natural-language classification approaches and fine-tuned for vulnerability detection.
- The work demonstrates the potential of machine learning to detect software vulnerabilities directly from source code.
- Future work should improve labels using dynamic-analysis findings or security patches so machine-learning scores become more complementary to static analysis.
- With larger and better-labeled datasets, deep learning for source-code analysis could become practical for more code-classification problems.The paper names style-violation detection, commit categorization, and algorithm or task classification as examples.