Source-linked AI summary
Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, Yang Liu
TL;DR
Vulnerability identification is difficult because insecure code creates exploitable risks and existing approaches struggle to represent diverse, complex program semantics. Devign learns vulnerability patterns from composite code graphs with a graph neural network and Conv module, outperforming baseline methods on manually labeled datasets from four diversified C libraries.
Problem
Vulnerability identification is challenging because vulnerabilities arise across diverse weakness types and libraries, making hand-crafted features impractical and existing deep-learning approaches incomplete in program semantics.
Method
Devign encodes a source-code function into a composite graph of multiple syntax and semantic representations, then applies gated graph neural layers and a Conv module for graph-level classification.
Results
10.51% higher average accuracy and 8.68% higher average F1 score than baseline methods on manually labeled datasets from four diversified C libraries.
Takeaways & Limitations
Devign achieves a new state of the art for machine-learning-based vulnerable-function discovery on real open-source projects.
Abstract
from arXiv · showhide
Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68\% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.
1 Introduction
Vulnerability identification is important but difficult because insecure code creates exploitable risks, while vulnerability diversity and heterogeneous program semantics limit hand-crafted and partial representations. Devign addresses this with composite code graphs and a graph neural network evaluated on manually labeled data from four diversified C libraries.
- Vulnerabilities in insecure code can be exploited to attack software systems and cause substantial financial and social damage.
- Vulnerability identification is challenging because weakness types and libraries vary, making hand-crafted features impractical across numerous libraries.
- Existing deep-learning approaches automate feature extraction but do not comprehensively learn the structural, logical, and heterogeneous semantics of complex real source code.
- Devign encodes multiple programming semantics in a composite graph and uses a graph neural network with a Conv module for graph-level vulnerability classification.The Conv module hierarchically selects coarser features from heterogeneous node representations.
- Devign is evaluated on manually labeled datasets collected from 4 popular and diversified C libraries.The datasets target complex, varied real source code rather than synthesized code.
- Devign achieves an average 10.51% higher accuracy and 8.68% higher F1 score than baseline methods.
2 The Devign Model
Devign represents each function as a composite code graph, propagates information across heterogeneous edges with a gated graph recurrent network, and uses a Conv module for graph-level vulnerability classification.
- Graph Embedding Layer of Composite Code Semantics: Devign maps function code into a joint graph combining AST, control-flow, data-flow, and sequential code representations.AST nodes form the shared node set, while heterogeneous edges encode the corresponding code relationships.
- Problem Formulation: The model performs graph-level classification to predict whether a function is vulnerable, using a sigmoid-based prediction layer over learned graph features.The paper formulates the task as learning a mapping from graph G to vulnerability label Y.
- Graph Embedding Layer of Composite Code Semantics: Each graph node is initialized from encoded source-code content and node type, using word2vec for code and label encoding for type.The two encodings are concatenated into the initial node representation.
- Gated Graph Recurrent Layers: A gated graph recurrent network propagates information through edge-type- and direction-dependent neighborhood aggregation to produce final node representations.The propagation repeats for T time steps, aggregates information across edge types, and uses GRUs to update node states.
- The Conv Layer: The Conv module selects task-relevant node and feature sets through 1-D convolution, max pooling, and dense layers before prediction.It is designed to avoid the limitations of globally gathering node embeddings and directly exploits the predefined ordering and connections in code graphs.
3 Evaluation
Devign is evaluated through comparisons with learning-based vulnerability predictors, Ggrn, composite and single-edged graphs, static analyzers, and recent CVE-derived vulnerable functions. Across these settings, the experiments report stronger performance for Devign and its Conv module.
- Evaluation Design: The evaluation compares Devign with learning-based predictors, Ggrn, static analyzers, and tests composite representations, single-edged graphs, and recent CVE-derived functions.The study uses accuracy and F1 score and evaluates manually labeled functions from four large C projects.
- Data Preparation: The dataset contains manually labeled functions from Linux Kernel, QEMU, Wireshark, and FFmpeg.Functions are extracted from vulnerability-fix and non-vulnerability-fix commits, then assigned corresponding labels.
- Performance Comparison: 10.51% relative accuracy gain and 8.68% relative F1 gain are reported for Devign against the baseline methods on average.The reported minimum relative F1 gains across Linux Kernel, QEMU, Wireshark, FFmpeg, and Combined are 2.31%, 11.80%, 6.65%, 4.04%, and 4.61%, respectively.
- Conv Module Ablation: 4.66% average accuracy gain and 6.37% average F1 gain are reported for Devign compared with Ggrn across all cases.The authors associate this gain with the Conv module extracting more related nodes and features for graph-level prediction.
- Code Representations: 2.69% average F1 improvement is reported for Devign’s composite graphs over single-edged graphs across all tests.Accuracy gains for composite graphs range from 0.11% to 3.75%, except that the Linux dataset does not follow the overall accuracy pattern.
- Static Analyzer Comparison: Devign outperforms the static analyzers by 27.99 F1 points on the imbalanced datasets containing 10% vulnerable functions.Cppcheck found zero vulnerabilities in three of the four single-project datasets.
- Recent Vulnerabilities: 74.11% average accuracy is achieved on 112 vulnerable functions derived from the latest 10 CVEs of each project.The authors present this result as evidence of Devign’s potential for identifying newly exposed vulnerabilities.
4 Related Work
Related work moves from automated neural models over flat source-code sequences toward structural representations that better express program logic and dependencies. Prior approaches include recurrent, convolutional, tree-based, and graph-based models for vulnerability detection and other program-analysis tasks.
- Sequence-Based Models: Early deep-learning vulnerability detectors treat source code as natural-language sequences and use LSTM, BiLSTM, or CNN architectures.These approaches apply natural-language-processing techniques to tokenized code or bag-of-words representations.
- Structural Models: Structural neural networks use tree or graph representations to express code logic and structure beyond flat sequences.Examples include gated graph recurrent networks for program verification, variable-related prediction, and binary code similarity.
5 Conclusion and Future Work
The paper introduces Devign as a composite-graph model for vulnerable-function discovery and reports state-of-the-art performance on real open-source projects. Future work targets large functions, cross-project detection, and explainable assessments.
- Conclusion: Devign encodes source-code functions into joint graphs combining multiple syntax and semantic representations for vulnerable-code discovery.The model leverages graph neural networks to learn from the composite graph representation.
- Future Work: The model achieves a new state of the art for machine-learning-based vulnerable-function discovery on real open-source projects.The stated future directions are program slicing for big functions, cross-project detection, and human-readable vulnerability assessment.