PaperlayerView exact revision on arXiv

Source-linked AI summary

SemVul: Semantic-Enhanced Graph Neural Networks for Code Property Graph-based Vulnerability Detection

Younas Affan, Leonardo Regano, Giorgio Giacinto

arXiv:2609.06035v1cs.SEcs.CR

TL;DR

Vulnerability detectors face limitations in capturing both program structure and code meaning. SemVul combines Code Property Graphs with pre-trained code semantics and evaluates the approach across benchmarks, reporting improved F1 and competitive or superior performance.

  • Problem

    Token-based models can miss program structure, while graph-based models may underuse semantic code content, leaving complementary representation gaps in vulnerability detection.

    Source: §1.1 · p. 2

  • Method

    SemVul combines Code Property Graph structure with pre-trained code embeddings and evaluates multiple GNN architectures for vulnerability detection.

    Source: §2.4 · p. 8, §3.4 · p. 12

  • Results

    Across four datasets, SemVul reports +10-25 F1 points from semantic embeddings and competitive or superior benchmark performance.

    Source: §6 · p. 26

  • Takeaways & Limitations

    The findings recommend prioritizing embeddings over architecture complexity, matching architecture to dataset characteristics, and expecting transfer penalties across datasets.

    Source: §5.8 · p. 26

    Token-based models omit program structure, while cited LLM evaluations report non-deterministic responses and high false-positive rates.

    Source: §1.1 · pp. 2–3

Abstract

from arXiv · show

Vulnerabilities in source code are often the root cause of cyberattacks worldwide, as attackers exploit weaknesses in software to gain unauthorized access, steal data, or disrupt services. In this study, we evaluated existing research approaches and propose SemVul, a vulnerability detection pipeline that demonstrates better generalization and higher accuracy in learning vulnerable code patterns. We propose a Code Property Graph-based vulnerability-detection approach combined with semantic-level enhancement, enabling the model to capture both the program's structural flow and the semantic meaning of the code. Our approach integrates both node-level and edge-level semantic embeddings using pre-trained code embedding techniques. We systematically evaluate multiple GNN architectures on publicly available benchmark datasets. SemVul is generic with respect to the programming language and supports multiple architectures. By integrating structural and semantic information, the proposed approach improves vulnerability detection performance. Our results show that SemVul outperforms existing approaches and provides better generalization.

1. Introduction

SemVul is motivated by the need for scalable, generalizable vulnerability detection that captures both program structure and code meaning. It combines Code Property Graphs with semantic embeddings and evaluates multiple architectures and benchmarks.

1. Introduction

  • Accurate, scalable early detection matters because source-code vulnerabilities can propagate across projects and contribute to severe security breaches.The introduction cites more than 25,000 new CVEs in 2024 and over 60 million new GitHub repositories created in 2020.Source: §1 · p. 2

1.1. Achievements and Limitations of Prior Work

  • Prior approaches leave complementary gaps: static and dynamic analysis have practical limitations, token models can miss program structure, and structural graph models can underuse code semantics.The introduction also describes brittleness in evaluated LLM approaches, including non-determinism, false positives, and sensitivity to variable renaming.Source: §1 · p. 2, §1.1 · pp. 2–3

1.2. Our Approach: Semantic-Enhanced Graph Learning

  • SemVul combines CPG structure with pre-trained code embeddings so GNNs can represent both program flow and code meaning.Its pipeline uses Joern-extracted graphs, semantic embeddings, edge-type-aware processing, and graph pooling before classification.Source: §1.2 · p. 3
  • The running buffer-overflow example shows untrusted input flowing through DFG edges to strcpy, while AST and CFG edges encode syntax and execution order.The example graph has seven nodes and three typed edge classes.Source: §1.2 · p. 4, Figure 2 · p. 4

1.3. Contributions

  • SemVul’s contributions include semantic-enhanced CPG learning, systematic architecture and embedding comparisons, a reproducible pipeline, benchmark results, and a language-agnostic design.The experiments focus on C/C++, while the described pipeline can extend to other Joern-supported languages by regenerating CPGs and retraining embeddings.Source: §1.3 · pp. 4–5

1.4. Research Questions

  • The research questions test embedding gains and architecture performance, then examine cross-dataset transfer and consistency across benchmarks with diverse characteristics.The transfer study trains on each dataset and evaluates on the others; the multi-dataset evaluation includes balanced, imbalanced, noisy-label, and clean-label settings.Source: §1.4 · p. 5

2. Background and Related Work

The related work traces vulnerability detection across program graphs, GNNs, code models, and hybrid approaches, while emphasizing dataset and evaluation challenges. SemVul builds on these lines by combining semantic code content with graph structure and comparing components across datasets.

2.1. Program Representations for Vulnerability Detection

  • Code Property Graphs unify syntax, control flow, and data dependencies in typed graphs, providing a representation that Joern can extract for downstream learning.Prior work including Devign used composite program graphs and GGNNs for function-level vulnerability classification.Source: §2.1 · p. 6

2.2. Graph Neural Networks for Code

  • GNNs learn from neighboring program-graph nodes, while heterogeneous operators and pooling variants can model typed relations and aggregate graph-level information.The literature includes GCN, GAT, GraphSAGE, GIN, and GGNN architectures, as well as hierarchical pooling methods.Source: §2.2 · pp. 6–7

2.3. Pre-trained Code Models

  • Recent code-representation research spans graph-based, fused, and LLM-guided systems, but the paper identifies CPG construction as shared across these approaches.It distinguishes SemVul by jointly reasoning over structure and precomputed node semantics, and by systematically evaluating multiple components rather than one fixed combination.Source: §2.3 · pp. 7–8
  • Pre-trained code models capture token semantics but may underuse explicit program structure; SemVul instead enriches CPG nodes with precomputed semantic embeddings before message passing.The paper positions its controlled comparisons across embedding models, relation-aware GNNs, and four benchmarks as a way to quantify component contributions.Source: §2.3 · pp. 7–8

2.4. Hybrid Approaches

  • Hybrid methods combine graph and language-model representations through strategies such as token-to-node mapping or late fusion, whereas SemVul uses early node-level concatenation.The paper motivates the fusion choice by noting that textual content can miss dependencies and structural models can miss semantic cues.Source: §2.4 · p. 8

2.5. Dataset Quality and Evaluation Challenges

  • Dataset duplicates, split contamination, class imbalance, schema differences, and graph-export mismatches complicate vulnerability-detection evaluation.The paper describes standardizing preprocessing, per-function splitting, graph extraction, and schema vocabularies to support cross-dataset comparisons.Source: §2.5 · p. 8

3. Methodology

SemVul frames vulnerability detection as binary classification over Code Property Graphs, combining structural node types with semantic code embeddings. Edge-type-aware GNNs process the graphs before pooling and classification.

3.1. Problem Formulation

  • Vulnerability detection is formulated as learning a function from a Code Property Graph to a binary prediction of whether its function contains a vulnerability.Source: §3.1 · p. 10

3.2. Code Property Graph Representation

  • Joern-generated CPGs unify AST, CFG, DFG, and PDG representations, capturing syntax, execution order, data dependencies, and program dependencies.Source: §3.2 · p. 10

3.3. Semantic-Enhanced Node Representation

  • Each CPG node concatenates a one-hot node-type encoding with a semantic embedding of its code content, making structure and semantics a joint feature.Source: §3.3 · pp. 10–11
  • Pretrained code models and dataset-trained Word2Vec provide fixed, precomputed embeddings; their general-purpose representations are hypothesized to correlate with vulnerability and tested in RQ1.Source: §3.3 · pp. 10–11
  • For strcpy versus strncpy, embeddings provide distinct inputs, while supervised GNN message passing combines call semantics with data-flow and control-flow context.Source: §3.3 · p. 11

3.4. Graph Neural Network Architectures

  • Four architectures test different treatments of typed edges: GGNN uses relation-specific weights and gated updates, RGAT adds relation attention, Multi-View GGNN separates views, and Call-Aware GNN prioritizes call nodes.Source: §3.4 · pp. 11–12

3.5. Graph-Level Pooling and Classification

  • Mean, max, and standard-deviation pooling capture average behavior, extremes, and graph variability; this multi-statistic pooling outperforms single-statistic pooling by 3+ F1 points.Source: §3.5 · p. 12

4. Experimental Setup

The experimental setup evaluates four C/C++ vulnerability benchmarks, standardizes their CPG data, and compares embedding and model configurations. Metrics are selected to reflect each research question and account for class imbalance.

4. Experimental Setup

  • Experiments used an NVIDIA RTX 5000 Ada GPU, fixed random seeds at 42, and weighted loss with pos_weight = nneg/npos to address class imbalance.Experiments used mixed precision where supported, and all completed within 24 hours per dataset.Source: §4 · p. 12

4.1. Datasets

  • Four benchmarks contain labeled C/C++ functions; PrimeVul has rigorous labeling and about 3.5% vulnerable functions, making it a highly imbalanced evaluation setting.Devign uses FFmpeg and Qemu functions, Draper includes CWE annotations, and Reveal was designed to address data-quality issues in earlier datasets.Source: §4.1 · pp. 12–13
  • Vulnerability prevalence ranges from about 45% in Devign to 3.5% in PrimeVul, while maximum function graph sizes reach 12,294 nodes in Devign and 18,682 in Reveal.Draper and Reveal have about 6% and 9% vulnerable functions, respectively; the section identifies PrimeVul as the most challenging and realistic setting.Source: §4.1 · p. 13

4.2. Data Preprocessing

  • The pipeline converts each split into a unified format, extracts CPGs with Joern, maps node and edge types, and combines one-hot type encodings with semantic embeddings.The resulting cached datasets support reproducible training across embedding and architecture configurations.Source: §4.2 · p. 14
  • CPG node and edge vocabularies vary by dataset, code patterns, and Joern version, so the pipeline discovers and validates schemas and synchronizes vocabulary indices.One-hot node-type dimensions differ across datasets; edge-aware models must adapt their weights, and cross-dataset transfer requires vocabulary mapping and retraining.Source: §4.2 · p. 14

4.3. Model Configurations

  • The experiments compare MLP and GNN architectures alongside multiple code-embedding strategies, with hidden dimension set to the total input feature size.That size is the node-type one-hot dimension plus the embedding dimension; training uses the same optimization setup across models.Source: §4.3 · pp. 14–15

4.4. Evaluation Metrics

  • RQ1 centers on ΔF1 for semantic enrichment; RQ2 selects architectures by F1, adds AUC, accuracy, and recall, and reports precision on PrimeVul.RQ3 measures transfer with F1 drop or retention, while RQ4 assesses F1 consistency across all four benchmarks. Because class imbalance can make accuracy misleading, the evaluation relies primarily on F1 and AUC.Source: §4.4 · p. 16

5. Results

The results test whether semantic embeddings improve vulnerability detection over structural-only CPG representations. Across datasets, adding code semantics improves F1, while richer discriminative information—not feature dimensionality alone—accounts for the gains.

5.1. RQ1: Impact of Semantic Embeddings

  • GraphCodeBERT achieves 66.01 F1 (+14.64) on Devign, while CodeT5+ reaches 60.47 F1 despite 272 dimensions, below Word2Vec-200d at 63.84 F1 with 216 dimensions.The fixed EdgeType-GNN comparison indicates that dimensionality alone does not explain the improvement; semantic information is the distinguishing factor.Source: Table 9 · p. 17, §5.1 · p. 18
  • Adding GraphCodeBERT improves F1 on all four datasets: +14.64 on Devign, +25.20 on Draper, +23.77 on Reveal, and +10.68 on PrimeVul.The comparison is against structural-only representations; the gains span datasets with different sizes, sources, and class balance.Source: §5.1 · p. 18, Table 10
  • Dataset-specific Word2Vec raises Devign F1 by +10.80 at 100 dimensions and +12.47 at 200; pre-trained models improve further, with GraphCodeBERT best at 66.01 F1 (+14.64).AUC follows the same ordering, and the smallest semantic representation already captures most of the improvement over structural-only features.Source: §5.1 · pp. 17–18
  • Structural-only representations perform poorly: F1 is 51.37 on Devign and 24–28 on Draper and Reveal, with high accuracy on imbalanced data masking missed vulnerable samples.Without code text, distinct calls such as strcpy and strncpy_s receive identical CALL-node encodings.Source: §5.1 · pp. 16–18
  • The findings support combining CPG structure with code semantics: topology alone lacks information needed to distinguish vulnerable from safe code.Semantic embeddings capture token co-occurrence or code meaning that one-hot node and edge encodings omit.Source: §5.1 · p. 18

5.2. RQ2: GNN Architecture Comparison

The architecture comparison finds consistent advantages for edge-type-aware models, although the best architecture varies by dataset. Cross-dataset transfer remains difficult, with substantial F1 degradation and better retention from MultiView GGNN.

5.2. RQ2: GNN Architecture Comparison

  • Edge-type-aware architectures consistently outperform edge-agnostic baselines, but the top model varies: EdgeType-GNN wins on Devign and Draper, while MultiView GGNN wins on Reveal.On Draper, EdgeType-GNN achieves 53.65 F1 versus 46.78 for vanilla GGNN.Source: §5.2 · p. 19, Table 15 · p. 20
  • PrimeVul’s 3.5% vulnerable rate, expert-reviewed labels, and deduplication make it a particularly challenging evaluation setting.Its cleaner labeling and extreme imbalance contextualize the lower F1 scores relative to other datasets.Source: §5.2 · pp. 19–20
  • On PrimeVul, the hypergraph architecture achieves 23.52 F1, while RGAT reaches the highest recall at 34.07% with 15.25% precision.All models exceed 94% accuracy, which reflects the majority class in this 3.5%-vulnerable dataset rather than detection capability.Source: §5.2 · p. 20

5.3. RQ3: Cross-Dataset Transfer

  • Cross-dataset transfer causes an average F1 drop of 14–37 points, and models trained on Devign have the largest average penalty (-37.13).The results attribute the challenge to distribution shift compounded by schema heterogeneity across datasets.Source: §5.3 · p. 20, Table 15 · p. 20
  • MultiView GGNN retains 78.5% performance in transfer, the best retention among the compared architectures.The transfer architecture comparison trains on Draper and tests on Reveal.Source: §5.3 · p. 20, Table 16 · p. 20

5.4. RQ4: Multi-Dataset Generalization

SemVul maintains competitive or state-of-the-art results across four benchmarks with varied class balance and label quality. The architectural analysis also favors shallow networks and multi-statistic pooling.

5.4. RQ4: Multi-Dataset Generalization

  • SemVul remains competitive or state-of-the-art across four datasets: 66.01 F1 on Devign, 53.65 on Draper, 48.60 on Reveal, and 23.52 on PrimeVul.The datasets span 3.5%–45.8% vulnerable samples and vary in label quality; Reveal’s result is below the reported 53.09 F1 for Vul-LMGNNs under a possibly different split.Source: Table 17 · p. 21, §5.4 · p. 21
  • On Devign, SemVul reaches 66.01 F1, +14.64 over structural-only baselines, though below the originally reported 73.26 F1 amid reproducibility concerns (Chakraborty et al. (2021)).On Draper, the result is 53.65 F1, +0.45 over the RNN baseline; on PrimeVul, 23.52 F1 exceeds basic fine-tuning code LM baselines at 18.05–21.43 F1.Source: §5.4 · p. 21

5.5. Analysis of Architectural Choices

  • Two-layer networks outperform deeper ones, while mean-plus-max-plus-standard-deviation pooling exceeds single-statistic approaches by more than 2.5 F1 points.The depth result aligns with the reported over-smoothing phenomenon as node representations converge with increasing depth.Source: §5.5 · p. 21

5.6. Comparison with Prior Work

Comparisons with prior work show strong results on Draper and Devign, competitive performance on Reveal, and an advantage over basic code-language-model fine-tuning baselines on PrimeVul. Some published results use different splits or protocols, limiting direct comparison.

5.6. Comparison with Prior Work

  • SemVul achieves 53.65 F1 on Draper, 0.45 above the best published RNN baseline, and 48.60 F1 on Reveal, 3.33 above VulBERTa.On Reveal, it remains below the 53.09 F1 reported by Vul-LMGNNs, potentially under a different split.Source: §5.6 · p. 21
  • On PrimeVul, SemVul reaches 23.52 F1, above basic fine-tuning code LM baselines at 18.05–21.43 F1.Other comparisons are not directly equivalent: LLMxCPG uses a balanced paired test set, and Vul-LMGNNs reports results on rebalanced Draper.Source: §5.6 · pp. 21–22
  • On Devign, SemVul’s 66.01 F1 exceeds GRACE’s 65.11 and Vul-LMGNNs’ 63.82 among directly comparable reports, but is below the originally reported 73.26 F1.The paper associates the gap with reproducibility concerns documented by Chakraborty et al. (2021).Source: §5.6 · p. 21

5.7. Computational Cost

CPG generation is dominated by a fixed startup cost, with a smaller, sub-linear cost increase as graph size grows. Model inference is much faster than CPG generation, and training and embedding costs are also reported.

CPG generation scaling

  • About 3.78 s of CPG-generation time is fixed overhead from launching a fresh JVM for Joern’s parse and export steps.The fitted slope adds 0.284 ms per node.Source: §5.7 · pp. 23–24
  • Spearman correlation with generation time is 0.84 for graph-node count versus 0.71 for source-line count, supporting graph complexity as a better cost predictor than raw source length.Source: §5.7 · p. 24
  • 3.78 s + 0.284 ms × V fit, with a 0.66 log–log exponent, shows sub-linear CPG-generation growth across 247 Devign functions.The largest graph (14,229 nodes, 99,704 edges) took 7.55 s to generate.Source: Table 22 · p. 23, §5.7 · p. 23, Figure 5 · p. 25

Testing cost

  • GPU inference stayed below 17 ms (about 0.2% of end-to-end testing time), while CPG generation took 3.3 to 7.6 s.The measured functions were classified one at a time at batch size 1.Source: §5.7 · pp. 24–25, Table 23 · p. 24

Embedding and training cost

  • Embedding ran at about 4,600 unique code strings per second, while Devign training took about 29 s per epoch and roughly 39 minutes per full run.Identical code strings were embedded once and cached for reuse across training runs.Source: §5.7 · p. 25, Table 24 · p. 25

5.8. Summary of Findings

The findings favor pre-trained embeddings over added architecture complexity, while emphasizing model depth, dataset characteristics, and transfer penalties as practical considerations.

Summary of findings

  • Pre-trained code embeddings provide larger gains than architectural innovations, supporting prioritization of embeddings over architecture complexity.The summary recommends shallow networks and matching architecture to dataset characteristics.Source: §5.8 · p. 26
  • Two-layer networks outperform deeper models while reducing training cost.Source: §5.8 · p. 26
  • MultiView GGNN handles class imbalance better, whereas EdgeType-GNN excels on balanced data.Source: §5.8 · p. 26
  • Cross-dataset transfer penalties should be expected; the paper recommends similar deployment data or domain-adaptation techniques.Source: §5.8 · p. 26

6. Conclusion

SemVul combines Code Property Graphs with pre-trained code embeddings and reports results across four datasets and six GNN architectures. It is positioned as an analyst aid, not an autonomous vulnerability decision-maker.

6. Conclusion

  • SemVul combines Code Property Graphs with pre-trained code embeddings, evaluated across four datasets and six GNN architectures.Source: §6 · p. 26
  • Across four datasets, graph topology alone yielded near-random performance, while semantic embeddings improved F1 by 10–25 points.SemVul reports 53.65 F1 on Draper, 48.60 F1 on Reveal, and 23.52 versus 18.05–21.43 F1 for code LM baselines on PrimeVul.Source: §6 · p. 26

6.1. Interpretability and Practical Use

  • SemVul is a detection aid: it flags candidate functions for analyst review, while human experts retain the final judgment on exploitability.The paper identifies explanation mechanisms as future work, including attention weights or subgraphs responsible for a prediction.Source: §6.1 · p. 26

Artifact Availability

The paper makes the materials needed to reproduce its pipeline available and points readers to the repository README for experiment-reproduction steps.

Artifact Availability

A. CPG Schema Details

Tables 26 and 27 catalog CPG node and edge types and their relevance to vulnerability detection; edge types are organized by source graph representation.

A. CPG Schema Details

Related papers

Loading 2609.06035v1…