Source-linked AI summary
SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, Zhaoxuan Chen
TL;DR
Software vulnerabilities remain difficult to detect broadly and early, while deep learning lacks program representations that capture vulnerability-relevant information. SySeVR provides a systematic syntax-, semantics-, and vector-based framework for C/C++ source code, and its evaluation detected 15 vulnerabilities absent from NVD across four software products.
Problem
Vulnerabilities remain prevalent, while deep learning's applicability to vulnerability detection is not systematically understood and programs lack natural vector representations.
Method
SySeVR automatically extracts syntax-based and semantics-based vulnerability candidates and vector representations incorporating syntax, data dependency, and control dependency.
Results
15 vulnerabilities not reported in NVD were detected across four software products; 7 were unknown and 8 were silently patched by vendors.
Takeaways & Limitations
SySeVR enables multiple neural-network types to detect various vulnerability kinds and makes deep neural networks, especially BGRU, more effective than prior detection methods.
Takeaways & Limitations
The study focuses on C/C++ source code and four vulnerability syntax characteristics covering 93.6% of collected SARD vulnerable programs, while SARD may not represent real-world products.
Abstract
from arXiv · showhide
The detection of software vulnerabilities (or vulnerabilities for short) is an important problem that has yet to be tackled, as manifested by the many vulnerabilities reported on a daily basis. This calls for machine learning methods for vulnerability detection. Deep learning is attractive for this purpose because it alleviates the requirement to manually define features. Despite the tremendous success of deep learning in other application domains, its applicability to vulnerability detection is not systematically understood. In order to fill this void, we propose the first systematic framework for using deep learning to detect vulnerabilities in C/C++ programs with source code. The framework, dubbed Syntax-based, Semantics-based, and Vector Representations (SySeVR), focuses on obtaining program representations that can accommodate syntax and semantic information pertinent to vulnerabilities. Our experiments with 4 software products demonstrate the usefulness of the framework: we detect 15 vulnerabilities that are not reported in the National Vulnerability Database. Among these 15 vulnerabilities, 7 are unknown and have been reported to the vendors, and the other 8 have been "silently" patched by the vendors when releasing newer versions of the pertinent software products.
1 INTRODUCTION
The paper addresses the difficulty of detecting diverse software vulnerabilities early without relying heavily on manually defined features. It proposes SySeVR, a systematic deep-learning framework for representing C/C++ programs with vulnerability-relevant syntax and semantics, and reports improved detection effectiveness and newly identified vulnerabilities.
- Software vulnerabilities remain prevalent, motivating early source-code detection despite continuing academic and industrial efforts.
- Existing similarity-based methods have high false-negative rates for vulnerabilities not caused by code cloning, while pattern-based methods are laborious and error-prone.
- SySeVR automatically extracts Syntax-based Vulnerability Candidates and Semantics-based Vulnerability Candidates to represent vulnerability-relevant syntax, data dependency, and control dependency.
- 126 types of vulnerabilities were collected from NVD and SARD to evaluate SySeVR.
- 30.4% average reduction in false-negative rate was achieved by adding data- and control-dependency information to SySeVR-enabled detectors.
- 15 vulnerabilities absent from NVD were detected across Libav, Seamonkey, Thunderbird, and Xen; 7 were unknown and 8 were silently patched.
2.1 Basic Idea and Framework Overview
SySeVR adapts the region-proposal idea by dividing programs into code pieces that expose vulnerability-related syntax and semantics. It transforms syntax-based candidates into semantically enriched candidates and encodes them as vectors for neural-network input.
- SySeVR is inspired by image-processing region proposals but targets program representations rather than naturally structured image regions.
- Treating an entire function as one region is too coarse because vulnerability detectors must locate vulnerable code, not only classify functions.
- The framework divides programs into smaller code pieces that exhibit vulnerability-related syntax and semantic characteristics.
- Figure 2 illustrates candidate extraction and representation, including nested SyVCs and the SyVC→SeVC transformation.
- A SyVC matches vulnerability syntax, a SeVC adds control- or data-dependent statements, and each SeVC is encoded as a vector for a deep neural network.
2.2 Extracting SyVCs
SyVC extraction identifies code elements whose syntax matches vulnerability characteristics. The procedure represents programs with functions, statements, tokens, and ASTs, then traverses each function's AST to collect matching elements.
- Vulnerability syntax characteristics, such as pointer usage, identify initial code candidates for vulnerability detection.
- The method uses AST-node attributes to describe vulnerability syntax characteristics because manually extracting vulnerable lines is extremely time-consuming.
- A program contains ordered functions, functions contain ordered statements, and statements contain tokens such as identifiers, operators, constants, and keywords.
- AST roots correspond to functions, leaves to tokens, and internal nodes to statements or consecutive token sequences that can represent SyVCs.
- Algorithm 1 generates each function's AST, traverses its code elements, matches them against vulnerability characteristics, and returns the SyVC set.
- A single code location may yield multiple SyVCs under different vulnerability syntax characteristics, and one SyVC may be part of another.
2.3 Transforming SyVCs to SeVCs
SyVCs are extended into SeVCs by incorporating statements semantically related through data and control dependencies. Program dependency graphs support this transformation and enable forward and backward slicing around each SyVC.
- SyVC→SeVC transformation: The SyVC→SeVC transformation uses program slicing to include statements semantically related to the original syntax-based candidate.The resulting representation accommodates information induced by data and control dependencies.
- Dependency graphs: Program dependency graphs represent data or control dependencies between pairs of statement or predicate nodes.A data dependency occurs when a computed value flows between nodes; control dependency is defined using post-dominance.
- Program slicing: Forward and backward slices are both considered because a SyVC may affect later statements or be affected by earlier statements.These two directions capture statements that may contain or contribute to a vulnerability.
Definition 7 (forward, backward, and program slices [27]
The framework constructs intra- and interprocedural forward and backward slices for each SyVC, merges them into an ordered program slice, and converts that slice into an ordered SeVC.
- Slice definitions: An intraprocedural forward slice contains nodes reachable from a SyVC, while a backward slice contains nodes from which the SyVC is reachable.Both slices are ordered subsets of the function’s program-dependency graph.
- Slice definitions: Interprocedural slices extend these definitions across function-call boundaries by collecting reachable or reaching nodes from multiple program-dependency graphs.Forward slicing follows called functions; backward slicing can include called and calling functions.
- Slice definitions: A program slice merges the interprocedural forward and backward slices at the SyVC while preserving order and omitting adjacent repeated nodes.The resulting ordered set contains nodes from the program-dependency graphs of functions in the program.
- Running example: In the running example, the program slice for SyVC “data” crosses func and println, producing an ordered SeVC with statements from both functions.The example SeVC lists statements from func before println because func calls println.
- Algorithm 2: Algorithm 2 generates program-dependency graphs, constructs intra- and interprocedural slices, and transforms each program slice into a SeVC.Statements are added according to their order within functions and their function-call ordering across functions.
2.4 Encoding SeVCs into Vectors
Algorithm 3 converts SeVCs into vectors while reducing dependence on user-defined names and preserving symbolic program information in fixed-length representations.
- Symbolic representation: Each SeVC is normalized by removing non-ASCII characters and comments, then mapping user-defined variables and functions to symbolic names.The mappings are one-to-one, using names such as “V1”, “V2”, “F1”, and “F2”.
- Vector encoding: The symbolic SeVC is lexically split into symbols, each symbol is mapped to a fixed-length vector, and concatenation produces one vector per SeVC.For example, “V1=V2-8;” becomes a sequence containing identifiers, operators, a number, and a delimiter.
- Length normalization: A threshold θ controls how encoded SeVC vectors are adjusted to a common length around the SyVC.The procedure retains sub-vectors on either side of the SyVC or deletes a leftmost portion when necessary.
2.5 Labeling SeVCs and Corresponding Vectors
SeVC vectors are labeled according to whether the represented SeVC contains a known vulnerability, producing binary labels for neural-network learning.
- Labeling: A SeVC containing a known vulnerability receives label “1”, while every other SeVC receives label “0”.The labels apply to both the SeVCs and their corresponding vectors.
3 EXPERIMENTS AND RESULTS
The experiments evaluate SySeVR across vulnerability types, neural networks, dependency information, and comparisons with existing methods. The framework uses vulnerability syntax characteristics to extract SyVCs and achieves broad coverage across the collected programs.
- SySeVR experiments address four questions about multi-kind detection, neural-network effectiveness, control dependency, and comparison with state-of-the-art methods.
- The vulnerability dataset combines NVD and SARD programs, including 1,591 open-source C/C++ programs from NVD and 14,000 C/C++ programs from SARD.
- 3.3.1 Extracting SyVCs: Checkmarx rules provide four vulnerability syntax characteristics: library/API calls, array usage, pointer usage, and arithmetic expressions.
- 3.3.1 Extracting SyVCs: The four characteristic types overlap across 126 CWE IDs, including 10 covered only by pointer-usage characteristics and 39 covered by all four types.
- 3.3.1 Extracting SyVCs: 420,627 SyVCs were extracted, covering 13,016 of 13,906 vulnerable SARD programs, or 93.6%.
- 3.3.4 Generating Ground-truth Labels of SeVCs: 56,395 SeVCs received vulnerable labels and 364,232 received non-vulnerable labels across the collected programs.
3.4 Experimental Results
Experiments show that SySeVR’s effectiveness depends on semantic program representations, context-preserving vectors, and suitable neural architectures. BGRU generally performs best, while semantic enrichment improves detection and the framework identifies vulnerabilities missed by existing methods.
- RQ1: 3.4% lower FPR and 5.0% lower FNR than VulDeePecker were achieved for library/API function-call vulnerabilities.SySeVR-BLSTM’s improvement is attributed to accommodating additional semantic information through SeVCs.
- RQ2: 4.5% lower FNR and 2.3% higher F1-measure were obtained by bidirectional RNNs over unidirectional RNNs on average.Bidirectional models can incorporate information from statements before and after the statement being classified.
- RQ2: BGRU was more effective than BLSTM, CNN, DBN, and shallow models, although all evaluated models had substantially higher FNRs than FPRs.The comparison used an FPR of 2.0% for each model.
- RQ2: 85.8% F1 and 83.7% MCC for word2vec-based BGRU exceeded the best bag-of-words results of 76.6% F1 and 73.7% MCC.Bag-of-words lacks context information; notably, BGRU reached 48.8% F1 and 46.9% MCC with that representation.
- RQ3: 30.4% lower FNR and 24.0% higher F1-measure were achieved by adding control dependency to data dependency on average.The experiments compared models using datasets with data dependency alone against models using both data and control dependencies.
- RQ4 and application: 15 vulnerabilities absent from NVD were detected across four software products, including 7 unknown vulnerabilities reported to vendors and 8 silently patched vulnerabilities.SySeVR-enabled BGRU also substantially outperformed state-of-the-art vulnerability detection methods in the reported comparison.
4 LIMITATIONS
The study identifies limitations in language scope, vulnerability coverage, dataset representativeness, labeling, and model design, while noting that some deep networks outperform existing methods.
- The framework targets C/C++ source code and may require adaptation for other programming languages or executables.
- Four vulnerability syntax characteristics cover 93.6% of vulnerable SARD programs, but SARD may not represent real-world software products.
- The SyVC and SeVC generation algorithms could be improved to incorporate more syntactic and semantic information.
- The experiments use one model for multiple vulnerability types, leaving separate specialized models versus one shared model for future study.
- The study reports 15 vulnerabilities detected by BGRU but not reported in the NVD, including 7 unknown and 8 silently patched vulnerabilities.
- Deep learning's success in vulnerability detection remains insufficiently explained, despite some models outperforming state-of-the-art methods.
5 RELATED WORK
Related work covers code similarity-based and pattern-based static detection, distinguishing rule-based and machine-learning approaches and positioning SySeVR among deep-learning methods.
- Source-code static vulnerability detectors are divided into code similarity-based and pattern-based methods.
- Pattern-based methods include rule-based tools with human-generated patterns and machine-learning methods using extracted vulnerability features.
- Rule-based tools often incur high false-positive and/or false-negative rates, while SySeVR learns vulnerability patterns automatically.
- Software-metric methods predict vulnerabilities coarsely, whereas vulnerable-code pattern recognition targets patterns from ASTs, code property graphs, or system calls.
- Deep-learning methods reduce manual feature definition and include function-level representation learning, slice-level VulDeePecker, and multiclass µVulDeePecker.
- SySeVR is presented as the first systematic framework using deep learning for vulnerability detection and as addressing weaknesses of VulDeePecker.
6 CONCLUSION
The paper presents SySeVR for deep-learning vulnerability detection, reports 15 vulnerabilities absent from the NVD, and identifies open problems for future research.
- SySeVR is a framework for using deep learning to detect software vulnerabilities.
- The authors report detecting 15 vulnerabilities that were not reported in the NVD.
- Among these findings, 7 were unknown and reported to vendors, while 8 had been silently patched in newer software versions.
- Future work includes addressing study limitations and investigating how code duplication affects SySeVR-enabled models.